1 1 Modern Data Architecture for Retail with Apache Hadoop on Windows A Hortonworks and Microsoft White Paper JUNE 2014
2 2 Executive Summary Retailers have a long history of investing in data and analytics technologies to drive more effective marketing, merchandising and operations. The recent explosion of new types and sources of data from the web and connected devices, social media, suppliers and elsewhere has created tremendous new opportunities for savvy retailers to boost sales, but also raises the bar: the race is on to harness this big data to keep pace with the competition. Retail firms that have adapted to the challenge of big data are using it to derive greater customer insights, promote increased brand engagement and loyalty, optimize pricing and promotions, streamline the entire supply chain, and develop innovative new business models. The most successful big data initiatives have integrated data from across many disparate data sources and siloes to achieve outsized results. Many retail enterprises have turned to Apache Hadoop to collect and manage diverse volumes of unstructured and semi-structured data alongside traditional repositories like the enterprise data warehouse. For these organizations, Hadoop is fulfilling the vision of an enterprise-wide repository for big data, frequently called a data lake. An enterprise data lake provides the following benefits to retailers: New efficiencies, through a significantly lower cost of storage and the optimization of data processing workloads such as data transformation and integration. New opportunities, through accelerated analytical applications, able to access all enterprise data in both batch and real-time modes. New insights, through allowing data from traditional and emerging data sources to be retained, combined and mined in new and unforeseen ways. Apache Hadoop provides these benefits through a technology core consisting of a scalable and flexible storage system that can accept data in any format, and an application framework that allows different types of processing workloads to interact with a common pool of stored data. This paper explores the role of Hadoop and its supporting ecosystem in meeting enterprise requirements for integrating big data with existing enterprise data systems as part of a modern data architecture. We pay particular attention to the capabilities of Apache Hadoop as a data platform and as an enabler of the enterprise data lake.
3 3 Disruption in the Data Corporate IT organizations in the retail industry have been tackling data challenges at scale for many years now. Traditional sources of data in retail firms include customer relationship management (CRM systems, enterprise resource planning (ERP systems, inventory management systems, transactional databases, product catalogs, and other systems supporting core enterprise functions. Shortly after these systems of record became established, the enterprise data warehouse (EDW emerged as a common repository for data extracted from these systems, allowing business intelligence applications to more easily tap into enterprise data. Today, every retailer has data warehouses that serve to model and capture the essence of the business from their enterprise systems. The Challenge of New Types of Data The emergence and explosion of new types of data in recent years has put tremendous pressure on all of the data systems within the enterprise. These new types of data stem from systems of engagement such as websites, or from the growth in connected devices. The data from these sources has a number of features that make it a challenge for a data warehouse: Exponential Growth. An estimated 2.8ZB of data in 2012 is expected to grow to 40ZB by Eighty-five percent of this data growth is expected to come from new types, with machinegenerated data being projected to increase 15x by (source: IDC Varied Nature. The incoming data can have little or no structure, or structure that changes too frequently for reliable schema creation at time of ingest. Find out more a bout these new types of da ta a t Hortonworks.com Clickstream Social Media Server Logs Geoloca tion Machine and Sensor Value at High Volumes. The incoming data can have little or no value as individual or small groups of records. But at high volumes or with a longer historical perspective, data can be inspected for patterns and used for advanced analytic applications. The Growth of Apache Hadoop Challenges of capture and storage aside, the value of blending existing enterprise data with these new types of data is being proven by many retailers. The technology that has emerged as the preferred way to realize this value is Apache Hadoop, whose momentum was described as unstoppable by Forrester Research in the Forrester Wave : Big Data Hadoop Solutions, Q The maturation of Apache Hadoop in recent years has broadened its capabilities from simple processing of large data sets to a full-fledged data platform offering the services necessary for enterprise adoption, including security, operations management and more. Wha t is Hadoop? Apache Hadoop is an opensource technology born out of the experience of web scale consumer companies such as Yahoo, Facebook and others, who were among the first to confront the need to store and process massive quantities of digital data.
4 4 Hadoop and Your Existing Data Systems: A Modern Data Architecture From an architectural perspective, the use of Hadoop as a complement to existing data systems is extremely compelling. An open source technology designed to run on large numbers of commodity servers, Hadoop provides a low-cost, scale-out approach to data storage and processing, and is proven to scale to meet the needs of the very largest retail organizations in the world. DATA*SYSTEMS* APPLICATIONS* Sta8s8cal Analysis Repositories BI/Repor8ng, AdHocAnalysis ROOMS RDBMS* EDW EDW* MPP MPP* Interac8veWeb &MobileApplica8ons ********************** Governance** &*Integra>on Data*Access Data*Management Enterprise Applica8ons Security Opera>ons DEV*&*DATA*TOOLS* Build&Test OPERATIONS*TOOLS* Provision, Manage& Monitor SOURCES* OLTP,ERP, CRMSystems Documents & s WebLogs, ClickStreams Social Networks Machine Generated Sensor Data GeoLloca8on Data Fig. 1 A Modern Data Architecture with Apache Hadoop integrated into existing data systems Hortonworks is dedicated to enabling Hadoop as a key component of the data center, and having partnered deeply with some of the largest data warehouse vendors we have observed several key opportunities and efficiencies Hadoop brings to the enterprise.
5 5 New Opportunities for Analytics The architecture of Hadoop offers new opportunities for data analytics: Schema On Read. Unlike an EDW, in which data is transformed into a specified schema when it is loaded into the warehouse requiring schema on write Hadoop empowers users to store data in its raw form. Analysts can then create a schema to suit the needs of their application or analysis at the time of use, enabling schema on read. Schema on read helps organizations overcome concerns about the lack of inherent structure in new data sources, and allows them to avoid the rigidity and waste that comes from pre-processing data when its ultimate use, format or value cannot be foreseen. For example, assume an application exists and combines CRM data with clickstream data to obtain a single view of a customer interaction. As new types of data become available and relevant (for example, server log or sentiment data, they too can be added to enrich the view of the customer. The key distinction being that at the time the data was stored, it was not necessary to declare its structure and association with any particular application. Hadoop:'A"New"Approach"to"Insight" Current'Approach' ' Apply"schema"on"write" " Heavily"dependent"on"IT" " Augment'with'Hadoop' ' Apply"schema"on"read" " Support"range"of"access"paJerns"to"data" stored"in"hdfs:"polymorphic"access" SQL' Single"Query"Engine" Repeatable"Linear"Process" Hadoop' Mul5ple"Query"Engines" Itera5ve"Process:"Explore,"Transform,"Analyze" Determine" list"of" ques5ons" Design" solu5ons" Collect" structured" data" Ask" ques5ons" from"list" Detect" addi5onal" ques5ons" Batch" Interac5ve" RealO5me" InOmemory" Fig. 2 Multi-use, Multi-workload Data Processing. By supporting multiple access methods (batch, realtime, streaming, in-memory, etc. to a common data set, Hadoop enables analysts to transform and view data in multiple ways (across various schemas, achieving closed-loop analytics and bringing timeto-insight closer to real time than ever before. For example, an online merchant may produce checkout recommendations in real-time based on a customer s actual shopping pattern, and run a series of batch processes overnight to produce generic product affinity recommendations. Hadoop enables this scenario to happen on a single cluster of shared resources and single versions of the data.
6 6 New Efficiencies for Data Architecture In addition to the opportunities for Big Data analytics, Hadoop offers efficiencies in a data architecture: Lower Cost of Storage. By design, Hadoop runs on low-cost commodity servers and direct-attached storage, allowing for a dramatically lower overall cost of storage. In particular, when compared to high-end Storage Area Networks (SANs from vendors such as EMC, scaling-out compute and storage using Hadoop provides an extremely compelling alternative. Hadoop allows the user to reduce capital expenditures both because it runs on commodity hardware and also because it allows users to invest in just enough hardware to meet immediate needs, and easily expand later as needs grow. The cost dynamic Hadoop offers makes it possible to store, process, analyze, and access more data than ever before. For example, in a traditional business intelligence application, it may have only been possible to access a single year of data after it was transformed from its original format and stored in the data warehouse. Using Hadoop, it becomes possible to retain 10 years of that data in its original format, in addition to the year stored in the EDW. This enables much richer applications with far greater historical context. Hadoop:'Lower&Cost&of&Storage& Cloud'Storage' HADOOP& Fully'Loaded'Cost'Per'Raw'TB'of'Data' (Min Max'Cost' NAS' Engineered'System' MPP' SAN' $0& $20,000& $40,000& $60,000& $80,000& $180,000& Fig. 3 Source: Juergen Urbanski, Board Member Big Data & Analytics, BITKOM
7 7 Data Warehouse Workload Optimization. The scope of tasks being executed by the EDW has grown considerably across ETL, analytics and operations. The ETL function is a relatively low-value computing workload that can be performed on Hadoop for a much lower cost. Many users offload this function to Hadoop, wherein data is extracted and transformed on the Hadoop cluster and the results are loaded into the data warehouse. This allows critical and high-cost EDW CPU cycles and storage space to be freed, allowing that system to perform the high-value analytics and operations functions that best leverage its advanced capabilities. Hadoop:'Data$Warehouse$Workload$Op1miza1on$ Current'Reality' ' EDW$at$capacity:$some$usage$from$$ low=value$workloads$ $ Older$transformed$data$archived,$$ unavailable$for$ongoing$explora1on$ $ Source$data$oCen$discarded$ $ Augment'with'Hadoop' ' Free$up$EDW$resources$from$low=value$tasks$ $ Keep$100%$of$source$data$and$historical$data$for$$ ongoing$explora1on$ $ Mine$data$for$value$aCer$loading$it$because$of$$ schema=on=read$ ANALYTICS 20% OPERATIONS 50% OPERATIONS 50% ANALYTICS 50% ETL PROCESS 30% Fig. 4
8 8 A Blueprint for Enterprise Hadoop As Apache Hadoop has become a mature and successful component of enterprise data architectures, the capabilities of the platform have expanded significantly in response to emerging requirements. For example, in its early days the components enabling storage (HDFS and compute (MapReduce represented the key elements of the Hadoop platform. While they remain crucial today, many supporting projects have been contributed to the Apache Software Foundation (ASF by both vendors and users alike, greatly expanding Hadoop s capabilities into a broader enterprise data platform. EnterpriseHadoop:Capabili'es* EnterpriseHadoopCapabili;es Presenta;on&Applica;ons Enable*both*exis'ng*and*new*applica'ons*to* provide*value*to*the*organiza'on** * EnterpriseManagement&Security Empower*exis'ng*opera'ons*and** security*tools*to*manage*hadoop* * GOVERNANCE& INTEGRATION DATAACCESS SECURITY OPERATIONS Load*data*and* manage*according** to*policy* * Access*your*data*simultaneously*in*mul'ple*ways* (batch,*interac've,*reald'me* Provide*layered* approach*to* security*through* Authen'ca'on,* Authoriza'on,* Accoun'ng,*and* Data*Protec'on* Deploy*and* effec'vely** manage*the* plaborm* * Store*and*process*all*of*your*Corporate*Data*Assets* DATAMANAGEMENT DeploymentChoice Provide*deployment*choice*across*physical,*virtual,*cloud* * Fig. 5 Hadoop s enterprise capabilities are aligned with the following functional areas, each a foundational requirement for any data platform technology: Data Management. Store and process vast quantities of data in a scale-out storage layer. Data Access. Access and interact with data in a wide variety of ways spanning batch, interactive, streaming, and real-time use cases. Data Governance & Integration. Quickly and easily load data, and manage it according to policy. Security. Address requirements of authentication, authorization, accounting and data protection. Operations. Provision, manage, monitor and operate Hadoop clusters at scale.
9 9 The Apache projects that perform these functions are detailed in the following diagram. This set of projects and technologies represent the core of Enterprise Hadoop. Key technology powerhouses such as Microsoft, SAP, Teradata, Yahoo, Facebook, Twitter, LinkedIn and many others are continually contributing to enhance the capabilities of the open source platform, each bringing their unique capabilities and use cases. As a result, the innovation of Enterprise Hadoop has continued to outpace all proprietary efforts. EnterpriseHadoop:Components Presenta:on&Applica:ons Enablebothexis1ngandnewapplica1onto providevaluetotheorganiza1on EnterpriseManagement&Security Empowerexis1ngopera1onsand securitytoolstomanagehadoop GOVERNANCE& INTEGRATION DATAACCESS SECURITY OPERATIONS DataWorkflow, Lifecycle& Governance Falcon Sqoop Flume WebHDFS Batch Map Reduce Script Pig SQL Hive/Tez HCatalog NoSQL Hbase Stream Storm YARN:DataOpera:ngSystem 1 HDFS (HadoopDistributedFileSystem Search Solr Others InCMemory Analy1cs ISVEngines n Authen:ca:on Authoriza:on Accoun:ng DataProtec:on Storage:HDFS Resources:YARN Access:Hive, Pipeline:Falcon Cluster:Knox Provision, Manage& Monitor Ambari Zookeeper Scheduling Oozie DATAMANAGEMENT DeploymentChoice Linux Windows OnCpremises Cloud Fig. 6 Data Management: HDFS provides Hadoop s efficient scale-out storage layer, and is designed to run across low-cost commodity hardware. YARN enables Hadoop to serve broad enterprise use cases, providing the resource management and pluggable architecture required to allow a wide variety of data access methods to operate on data stored in Hadoop with predictable performance and service levels. Data Access: Apache Hive offers direct data connections to Microsoft Excel and Power BI, and is the most widely adopted data access technology, though there are many specialized engines. For instance, Apache Pig provides scripting capabilities, Apache Storm offers real-time processing, and Apache HBase offers columnar NoSQL storage. All of these engines can work across one set of data and resources thanks to YARN. YARN also provides flexibility for new and emerging data access methods, including search, and programming frameworks such as Cascading.
10 10 Data Governance & Integration: Apache Falcon provides policy-based workflows for governance, while Apache Flume and Sqoop enable easy data ingestion, as do the NFS and WebHDFS interfaces to HDFS. Security: Security is provided at every layer of the Hadoop stack, from HDFS and YARN to Hive and the other Data Access components, on up through the entire perimeter of the cluster via Apache Knox. Operations: Apache Ambari offers the necessary interface and APIs to provision, manage and monitor Hadoop clusters and integrate with other management console software. A Thriving Ecosystem Beyond these core components, and as a result of innovations such as YARN, Apache Hadoop has a thriving ecosystem of vendors providing additional capabilities and/or integration points. These partners contribute to and augment Hadoop with given functionality, and this combination of core technology and ecosystem support provides compelling solutions for enterprises whatever their use case. Examples of partner integrations include: Business Intelligence and Analytics: All of the major BI vendors offer Hadoop integration, and specialized analytics vendors offer niche solutions for specific data types and use cases. Microsoft integrations with Excel and Power BI allow users to take advantage of Hadoop-based data and analytics in the familiar Office 365 environment. Data Management and Tools: There are many partners offering vertical and horizontal data management solutions alongside Hadoop, and there are numerous tool sets from SDKs to full IDE experiences for developing Hadoop solutions. Hortonworks and Microsoft have a deep and broad ecosystem of partners, and strategic relationships with key data center vendors: HP Rackspace Red Hat SAP Dell Infrastructure: While Hadoop is designed for commodity hardware, it can also run as an appliance, and be easily integrated into other storage, data and management solutions both on-premises and in the cloud. Systems Integrators: Naturally, given its success as a component of a full-featured enterprise data architecture, SIs of all sizes are building skills to assist with integration and solution development. As many of these vendors are already present within the enterprise providing similar capabilities for data warehouses, implementation risk is mitigated as existing tools and skills from EDW workloads are applied to Enterprise Hadoop. There is also a thriving ecosystem of new vendors emerging on top of the Enterprise Hadoop platform. These new companies are taking advantage of open APIs and new platform capabilities to create an entirely new generation of applications. The applications they re building leverage both existing and new types of data and are performing new types of processing and analysis that weren t technologically or financially feasible before the emergence of Hadoop. The result is that these new businesses are harnessing the massive growth in data creating opportunities for improved insight into customers, better medical research and healthcare delivery, more efficient energy exploration and production, predictive policing and much more.
11 11 Toward the Enterprise Data Lake Implementing Hadoop as part of an enterprise data architecture is a substantial decision for any enterprise. While Hadoop s momentum is unstoppable (according to Forrester, its adoption is a journey from single-instance applications to a full-fledged data lake. This journey has been observed many times across our customer base. New Analytics Applications Hadoop usage most typically begins with the desire to create new analytics applications fueled by data that was not previously being captured. While the specific application will be invariably unique to an industry or organization, there are many similarities between the types of data. Examples of analytics applications across industries include: Fig. 7 INDUSTRY USECASE DATATYPE FinancialServices NewAccountRiskScreens Telecom Sensor ServerLogs TradingRisk InsuranceUnderwri7ng CallDetailRecords(CDR InfrastructureInvestment Text Retail 360 ViewoftheCustomer Localized,PersonalizedPromo7ons WebsiteOp7miza7on Manufacturing SupplyChainandLogis7cs Social AssemblyLineQualityAssurance Healthcare UseGenomicDatainMedialTrials PharmaceuGcals MonitorPa7entVitalsinRealTime RecruitandRetainPa7entsforDrug Trials Geographic Machine Clickstream ImprovePrescrip7onAdherence Oil&Gas UnifyExplora7on&Produc7onData Government MonitorRigSafetyinRealTime ETLOffloadedResponsetoFederal BudgetaryPressures Sen7mentAnalysisforGovernment Programs Structured Unstructured Enterprise Hadoop Read about other industry use cases Healthcare Telecommunications Retail Manufacturing Oil & Gas Advertising Government
12 12 Increases in Scope and Scale As Hadoop proves its value in one or more application instances, increased scale or scope of data and operations is applied. Gradually, the resulting data architecture assists an organization across many applications. The case studies later in this paper describe the journeys taken by retail customers in pursuit of a data lake. Moving'Toward'A'Data'Lake Scale' A'Modern'Data'Architecture' ' '''''''''''''''''''''' RDBMS' MPP' EDW' Governance'' &'Integra;on' Data'Access Data'Management Security Opera;ons New'Analy;cs'Apps' Newtypesofdata LOB0driven Scope' Fig. 8
13 13 Vision of a Data Lake With continued growth in the scope and scale of analytics applications using Hadoop within the enterprise, the vision of an enterprise data lake can become a reality. In a practical sense, a data lake is characterized by three key attributes: Collect Everything: A data lake contains all data, both raw sources over extended periods of time as well as any processed data. Dive In Anywhere: A data lake enables users across multiple business units to refine, explore and enrich data on their terms. Flexible Access: A data lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines. The result: A data lake delivers maximum scale and insight with the lowest possible friction and cost. As data continues to grow exponentially, Enterprise Hadoop and EDW investments can provide a strategy for both efficiency in a modern data architecture and opportunity in an enterprise data lake. DATA*SYSTEMS* APPLICATIONS* Sta8s8cal Analysis Repositories BI/Repor8ng, AdHocAnalysis ROOMS RDBMS* EDW EDW* MPP MPP* Interac8veWeb &MobileApplica8ons ********************** Governance** &*Integra>on Data*Access Data*Management Enterprise Applica8ons Security Opera>ons DEV*&*DATA*TOOLS* Build&Test OPERATIONS*TOOLS* Provision, Manage& Monitor SOURCES* OLTP,ERP, CRMSystems Documents & s WebLogs, ClickStreams Social Networks Machine Generated Sensor Data GeoLloca8on Data Fig. 9 A Modern Data Architecture with Apache Hadoop integrated into existing data systems
14 14 Hadoop in Retail When Hadoop is integrated with modern retail operations, it dramatically reduces the cost of capturing, ingesting, storing and analyzing data. This enables retailers to analyze enough data to make statistically confident observations on empirical retail data, rather than rolling the dice with customer panels, small samples and focus groups, to guess what drives sales. Hadoop&for&Retail& ANALYSIS& Recommenda*onengine Brandhealth Pricesensi*vity Top8downclustering Productmix Webpathop*miza*on A/Btes*ng & DATA&& REPOSITORIES& & &&&&&&&&&&&&&&&&&&&&&& EDW& ERP& CRM& RDBMS& Governance& &&IntegraGon& Batch& & & Script& SQL& NoSQL& Stream& Search& & & & & & & & & & & YARN&:&Data&OperaGng&System 1 HDFS&& Others& (HadoopDistributedFileSystem N Security& OperaGons& TRADITIONAL&SOURCES& CRM& WEB&TRANSACTIONS& & PRODUCT&CATALOG& STAFFING& & CLICKSTREAM& WIFI&LOGS& SENSOR&RFID& ERP& POS&TRANSACTIONS& INVENTORY& STORES& SERVER&LOGS& SOCIAL&MEDIA& LOCATIONS&DATA& Fig. 10
15 15 Case Studies Build a 360 View of the Customer Retailers interact with customers across multiple channels, yet customer interaction and purchase data is often isolated in disparate silos. Few retailers can accurately correlate eventual customer purchases with marketing campaigns and online browsing behavior. Apache Hadoop gives retailers a 360 view of customer behavior. It lets them store data longer and apply interactive and batch analytics to identify the various phases of the customer lifecycle. Better customer analytics leads to greater customer insight, which helps increase sales and loyalty, reduce inventory expenses and retain the best customers. Analyze Brand Sentiment Enterprises retailers lack a reliable way to track and analyze brand health. It s difficult to explore the impact of advertising, competitor moves, product launches and news stories on a brand. Canned social media dashboards are not enough, and internal brand studies can be slow, expensive and flawed. Apache Hadoop enables quick, unbiased snapshots of brand opinions expressed in social media. Retailers can analyze sentiment from Twitter, Facebook, LinkedIn or industry-specific social media streams. With a better understanding of customer perceptions, retailers can align their communications, products and promotions with those perceptions. Localize and Personalize Promotions Retailers able to harness the converging forces of social, mobile and local are well positioned to win with the modern consumer. Merchants that can geo-locate their mobile subscribers can deliver localized and personalized promotions. This requires connections with both historical and real-time streaming data. Apache Hadoop brings the data together to inexpensively localize and personalize promotions delivered to mobile devices. Retailers can develop mobile apps to notify customers about local events and sales that align with their preferences and geographic location even down to a particular section in a specific store.
16 16 Build a Modern Data Architecture with Enterprise Hadoop and Microsoft Windows To realize the value in your investment in big data, use the blueprint for Enterprise Hadoop to integrate with your EDW and related data systems. Building a modern data architecture enables your organization to store and analyze the data most important to your business at massive scale, extract critical business insights from all types of data from any source, and ultimately improve your competitive position in the market and maximize customer loyalty and revenues. Read more at APPLICATIONS* MicrosoF*Applica=ons* New%% Power*BI* DEV*&*DATA*TOOLS* DATA**SYSTEM* MicrosoF*Analy=cs*PlaHorm* System* HDInsight* Azure* OPERATIONAL*TOOLS* INFRASTRUCTURE* SOURCES* Exis=ng*Sources** (CRM,*ERP,*Clickstream,*Logs* Emerging*Sources** (Sensor,*Sen=ment,*Geo,* Unstructured* Fig. 11
17 17 Hortonworks Data Platform for Windows Provides Enterprise Hadoop Hortonworks Data Platform (HDP for Windows is powered by 100% Open Source Apache Hadoop. HDP for Windows provides all of the Apache Hadoop-related projects necessary to integrate Hadoop into a Windows environment as part of a modern data architecture. Hortonworks Data Platform GOVERNANCE& INTEGRATION DATAACCESS SECURITY OPERATIONS DataWorkflow, Lifecycle& Governance Falcon Sqoop Flume NFS WebHDFS Batch Map Reduce Script Pig SQL Hive/Tez HCatalog NoSQL HBase Accumulo Stream Storm YARN:DataOpera9ngSystem 1 HDFS (HadoopDistributedFileSystem Search Solr Others In>Memory ISVEngines n Authen9ca9on Authoriza9on Accoun9ng DataProtec9on Storage:HDFS Resources:YARN Access:Hive, Pipeline:Falcon Cluster:Knox Provision, Manage& Monitor Ambari Zookeeper Scheduling Oozie DATAMANAGEMENT Fig. 12
18 18 Three key values: HDP for Windows provides an enterprise with three key values: Completely Open HDP provides Apache Hadoop for the enterprise, developed completely in the open, and supported by the deepest technology expertise. HDP incorporates the most current community innovation and is tested on the most mature Hadoop test suite and on thousands of nodes. HDP is developed and supported by engineers with the deepest and broadest knowledge of Apache Hadoop. Fundamentally Versatile HDP is designed to meet the changing needs of big data processing within a single platform while providing a comprehensive platform across governance, security and operations. HDP supports all big data scenarios, including batch, interactive, real time and streaming. HDP offers a versatile data access layer through YARN at the core of enterprise Hadoop that allows new processing engines to be incorporated as they become ready for enterprise consumption. HDP provides the comprehensive enterprise capabilities of security, governance and operations for enterprise implementation of Hadoop. Wholly Integrated HDP is designed to run in any data center and integrates with any existing system. HDP for Windows can be deployed in any Windows environment, onpremises or in the cloud, and is fully compatible with Linux-based HDP deployments. Developed in partnership with Microsoft, HDP for Windows is deeply integrated with Microsoft tools, technologies and applications, including.net, Microsoft Office, Power BI, SQL Server and the Analytics Platform System. Components of Enterprise Hadoop Read more about the individual components of Enterprise Hadoop. Da ta Management hdfs yarn Da ta Access mapreduce pig hive tez hbase storm hcatalog Da ta Governa nce & Integra tion falcon flume Security knox security Opera tions ambari HDP for Windows Deployment Options HDP for Windows offers multiple deployment options: On-premises: HDP for Windows is the only Hadoop platform that works seamlessly in the Windows environment, and is easily deployed on any commodity hardware. Cloud: HDP for Windows can be run in the Microsoft Azure IaaS environment, and is fully compatible with the Microsoft Azure HDInsight Service. Appliance: The Microsoft Analytics Platform System is a turnkey big data analytics appliance combining a high performance MPP SQL Server data warehouse with HDP for Windows in a single fully-integrated solution.
19 19 On-Premises Cloud HDP$on$Windows$ HDP$on$Linux$ FullcontrolofHWand so0wareconfigs 1 2 HDInsight Managed Hadoop Service Built on Azure storage Analytics Platform System Turnkey Hadoop and relational warehouse appliance 3 4 HDP on Windows HDP on Linux Your deployment of Hadoop hosted as a VM in Azure Fig. 13 Why Hortonworks for Hadoop? Founded in 2011 by 24 engineers from the original Yahoo Hadoop development and operations team, Hortonworks has amassed more Hadoop experience under one roof than any other organization. Our team members are active participants and leaders in Hadoop development; designing, building and testing the core of the Hadoop platform. We have years of experience in Hadoop operations and are best suited to support your mission-critical Hadoop project. Read more at Open Leadership Ecosystem Endorsement Enterprise Rigor Hortonworks has a singular focus and commitment to drive innovation in the open exclusively via the Apache Software Foundation process. Hortonworks is responsible for the majority of core code-base advances to deliver Apache Hadoop as an enterprise data platform. Hortonworks is focused on the deep integration of Hadoop with existing data center technologies and team capabilities. Hortonworks has secured strategic relationships with trusted data center partners including Microsoft, SAP, Teradata, Rackspace, and many more. Hortonworks has a world-class enterprise support and services organization with vast experience in the largest Hadoop deployments. Hortonworks engineers and certifies Apache Hadoop with the enterprise in mind, all tested with real-world rigor in the world s largest Hadoop clusters. For an independent analysis of Hortonworks Data Platform, you can download the Forrester Wave : Big Data Hadoop Solutions, Q from Forrester Research.
20 20 About Hortonworks Hortonworks develops, distributes and supports the only 100% open source Apache Hadoop data platform. Our team comprises the largest contingent of builders and architects within the Hadoop ecosystem who represent and lead the broader enterprise requirements within these communities. The Hortonworks Data Platform provides an open platform that deeply integrates with existing IT investments and upon which enterprises can build and deploy Hadoop-based applications. Hortonworks has deep relationships with the key strategic data center partners that enable our customers to unlock the broadest opportunities from Hadoop. For more information, visit About Microsoft Founded in 1975, Microsoft (Nasdaq: MSFT is the worldwide leader in software, services and solutions that help people and businesses realize their full potential.
Modern Data Architecture with Apache Hadoop Talend Big Data Presented by Hortonworks and Talend Executive Summary Apache Hadoop didn t disrupt the datacenter, the data did. Shortly after Corporate IT functions
1 Modern Data Architecture for Financial Services with Apache Hadoop on Windows A Hortonworks and Microsoft White Paper JUNE 2014 2 Executive Summary Financial services firms have long been dependent on
HDP Hadoop From concept to deployment. Ankur Gupta Senior Solutions Engineer Rackspace: Page 41 27 th Jan 2015 Where are you in your Hadoop Journey? A. Researching our options B. Currently evaluating some
HDP Enabling the Modern Data Architecture Herb Cunitz President, Hortonworks Page 1 Hortonworks enables adoption of Apache Hadoop through HDP (Hortonworks Data Platform) Founded in 2011 Original 24 architects,
Modern Data Architecture with Enterprise Apache Hadoop Hortonworks. We do Hadoop. Jeff Markham Technical Director, APAC firstname.lastname@example.org Page 1 Our Mission: Enable your Modern Data Architecture
Hadoop, the Data Lake, and a New World of Analytics Hortonworks. We do Hadoop. Spring 2014 Version 1.0 Page 1 Hortonworks Inc. 2014 Traditional Data Architecture Pressured 2.8 ZB in 2012 85% from New Data
A Modern Data Architecture with Apache Hadoop A Hortonworks White Paper March 2014 Executive Summary Apache Hadoop didn t disrupt the datacenter, the data did. Shortly after Corporate IT functions within
Big Data Realities Hadoop in the Enterprise Architecture Paul Phillips Director, EMEA, Hortonworks email@example.com +44 (0)777 444 3857 Hortonworks Inc. 2012 Page 1 Agenda The Growth of Enterprise
Hortonworks Data Platform Buyer s Guide Hortonworks Data Platform (HDP Completely Open and Versatile Hadoop Data Platform 2 2014 Hortonworks, Inc. All rights reserved. Hadoop and the Hadoop elephant logo
THE JOURNEY TO A DATA LAKE 1 THE JOURNEY TO A DATA LAKE 85% OF DATA GROWTH BY 2020 WILL COME FROM NEW TYPES OF DATA ACCORDING TO IDC, AS MUCH AS 85% OF DATA GROWTH BY 2020 WILL COME FROM NEW TYPES OF DATA,
WHITEPAPER OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT A top-tier global bank s end-of-day risk analysis jobs didn t complete in time for the next start of trading day. To solve
Big Data Analytics for Retail with Apache Hadoop A Hortonworks and Microsoft White Paper 2 Contents The Big Data Opportunity for Retail 3 The Data Deluge, and Other Barriers 4 Hadoop in Retail 5 Omni-Channel
The Future of Data Management with Hadoop and the Enterprise Data Hub Amr Awadallah (@awadallah) Cofounder and CTO Cloudera Snapshot Founded 2008, by former employees of Employees Today ~ 800 World Class
A Modern Data Architecture with Apache Hadoop A Hortonworks White Paper March 2014 Executive Summary Apache Hadoop didn t disrupt the datacenter, the data did. Shortly after Corporate IT functions within
Hortonworks & SAS Analytics everywhere. Page 1 A change in focus. A shift in Advertising From mass branding A shift in Financial Services From Educated Investing A shift in Healthcare From mass treatment
Session 0202: Big Data in action with SAP HANA and Hadoop Platforms Prasad Illapani Product Management & Strategy (SAP HANA & Big Data) SAP Labs LLC, Bellevue, WA Legal disclaimer The information in this
Data Security in Hadoop Eric Mizell Director, Solution Engineering Page 1 What is Data Security? Data Security for Hadoop allows you to administer a singular policy for authentication of users, authorize
Architecting the Future of Big Data Whitepaper Apache Hadoop: The Big Data Refinery Introduction Big data has become an extremely popular term, due to the well-documented explosion in the amount of data
Microsoft Big Data Solution Brief Contents Introduction... 2 The Microsoft Big Data Solution... 3 Key Benefits... 3 Immersive Insight, Wherever You Are... 3 Connecting with the World s Data... 3 Any Data,
Modernizing Your Data Warehouse for Hadoop Big data. Small data. All data. Audie Wright, DW & Big Data Specialist Audie.Wright@Microsoft.com O 425-538-0044, C 303-324-2860 Unlock Insights on Any Data Taking
WHITEPAPER RED HAT AND HORTONWORKS: OPEN MODERN DATA ARCHITECTURE FOR THE ENTERPRISE A Hortonworks and Red Hat whitepaper INTRODUCTION WHAT IS HADOOP? Apache Hadoop is an opensource technology born out
Community Driven Apache Hadoop Apache Hadoop Patterns of Use April 2013 2013 Hortonworks Inc. http://www.hortonworks.com Big Data: Apache Hadoop Use Distilled There certainly is no shortage of hype when
Datenverwaltung im Wandel - Building an Enterprise Data Hub with Cloudera Bernard Doering Regional Director, Central EMEA, Cloudera Cloudera Your Hadoop Experts Founded 2008, by former employees of Employees
Modern Data Architecture for Predictive Analytics David Smith VP Marketing and Community - Revolution Analytics John Kreisa VP Strategic Marketing- Hortonworks Hortonworks Inc. 2013 Page 1 Your Presenters
The Future of Data Management with Hadoop and the Enterprise Data Hub Amr Awadallah Cofounder & CTO, Cloudera, Inc. Twitter: @awadallah 1 2 Cloudera Snapshot Founded 2008, by former employees of Employees
Harnessing big data with Hortonworks Data Platform and Red Hat JBoss Data Virtualization Kimberly Palko, Product Manager Red Hat JBoss Doug Reid, Director Partner Product Management Hortonworks Cojan van
Community Driven Apache Hadoop Apache Hadoop Basics May 2013 2013 Hortonworks Inc. http://www.hortonworks.com Big Data A big shift is occurring. Today, the enterprise collects more data than ever before,
The Digital Enterprise Demands a Modern Integration Approach Nada daveiga, Sr. Dir. of Technical Sales Tony LaVasseur, Territory Leader Yesterday s approach to data and application integration is a barrier
Hortonworks CISC Innovation day Simon gregory firstname.lastname@example.org Here was the ask Hortonworks' data reposition - how this works and the types of data you work with. 1: Data Types & Value. What have
Please give me your feedback Session BB4089 Speaker Claude Lorenson, Ph. D and Wendy Harms Use the mobile app to complete a session survey 1. Access My schedule 2. Click on this session 3. Go to Rate &
Cloudera Enterprise Data Hub in Telecom: Three Customer Case Studies Version: 103 Table of Contents Introduction 3 Cloudera Enterprise Data Hub for Telcos 4 Cloudera Enterprise Data Hub in Telecom: Customer
Using Tableau Software with Hortonworks Data Platform September 2013 2013 Hortonworks Inc. http:// Modern businesses need to manage vast amounts of data, and in many cases they have accumulated this data
Bringing Big Data to People Microsoft s modern data platform SQL Server 2014 Analytics Platform System Microsoft Azure HDInsight Data Platform Everyone should have access to the data they need. Process
Big Data: Making Sense of it all! Jamie Engesser E-mail : email@example.com Page 1 Data Driven Business? Facts not Intuition! Data driven decisions are better decisions its as simple as that. Using
IIS AND HORTONWORKS ENABLING A NEW ENTERPRISE DATA PLATFORM 1 IIS AND HORTONWORKS SEEK TO ENABLE A NEW ENTERPRISE DATA PLATFORM TO POWER AN INNOVATIVE, MODERN DATA ARCHITECTURE IIS AND HORTONWORKS ENABLING
David Chappell SELLING PROJECTS ON THE MICROSOFT BUSINESS ANALYTICS PLATFORM A PERSPECTIVE FOR SYSTEMS INTEGRATORS Sponsored by Microsoft Corporation Copyright 2014 Chappell & Associates Contents Business
White Paper The Business Analyst s Guide to Hadoop Get Ready, Get Set, and Go: A Three-Step Guide to Implementing Hadoop-based Analytics By Alteryx and Hortonworks (T)here is considerable evidence that
Integrating Cloudera and SAP HANA Version: 103 Table of Contents Introduction/Executive Summary 4 Overview of Cloudera Enterprise 4 Data Access 5 Apache Hive 5 Data Processing 5 Data Integration 5 Partner
Aligning Your Strategic Initiatives with a Realistic Big Data Analytics Roadmap 3 key strategic advantages, and a realistic roadmap for what you really need, and when 2012, Cognizant Topics to be discussed
Capitalize on Big Data for Competitive Advantage with Bedrock TM, an integrated Management Platform for Hadoop Data Lakes Highly competitive enterprises are increasingly finding ways to maximize and accelerate
End to End Solution to Accelerate Data Warehouse Optimization Franco Flore Alliance Sales Director - APJ Big Data Is Driving Key Business Initiatives Increase profitability, innovation, customer satisfaction,
White Paper Protecting Big Data Data Protection Solutions for the Business Data Lake Abstract Big Data use cases are maturing and customers are using Big Data to improve top and bottom line revenues. With
Value 10/06/2015 2015 MapR Technologies 2015 MapR Technologies 1 Information Builders Mission & Value Proposition Economies of Scale & Increasing Returns (Note: Not to be confused with diminishing returns
VIEWPOINT High Performance Analytics Industry Context and Trends In the digital age of social media and connected devices, enterprises have a plethora of data that they can mine, to discover hidden correlations
Talend Big Data Delivering instant value from all your data Talend 2014 1 I may say that this is the greatest factor: the way in which the expedition is equipped. Roald Amundsen race to the south pole,
BIG DATA: FROM HYPE TO REALITY Leandro Ruiz Presales Partner for C&LA Teradata Evolution in The Use of Information Action s ACTIVATING MAKE it happen! Insights OPERATIONALIZING WHAT IS happening now? PREDICTING
IBM InfoSphere BigInsights Enterprise Edition Efficiently manage and mine big data for valuable insights Highlights Advanced analytics for structured, semi-structured and unstructured data Professional-grade
Hadoop Introduction Olivier Renault Solution Engineer - Hortonworks Hortonworks A Brief History of Apache Hadoop Apache Project Established Yahoo! begins to Operate at scale Hortonworks Data Platform 2013
Hortonworks Data Platform for Hadoop and SAP HANA Prasad illapani, Big Data & SAP HANA- Product Management & Strategy SAP Labs LLC., Bellevue, WA Bob Page, VP Partner Products, Hortonworks Inc. Palo Alto,
Data Services Advisory Modern Datastores An Introduction Created by: Strategy and Transformation Services Modified Date: 8/27/2014 Classification: DRAFT SAFE HARBOR STATEMENT This presentation contains
Tap into Hadoop and Other No SQL Sources Presented by: Trishla Maru What is Big Data really? The Three Vs of Big Data According to Gartner Volume Volume Orders of magnitude bigger than conventional data
White Paper BIG DATA-AS-A-SERVICE What Big Data is about What service providers can do with Big Data What EMC can do to help EMC Solutions Group Abstract This white paper looks at what service providers
Collaborative Big Data Analytics 1 Big Data Is Less About Size, And More About Freedom TechCrunch!!!!!!!!! Total data: bigger than big data 451 Group Findings: Big Data Is More Extreme Than Volume Gartner!!!!!!!!!!!!!!!
Investor Presentation Second Quarter 2015 Note to Investors Certain non-gaap financial information regarding operating results may be discussed during this presentation. Reconciliations of the differences
June, 2012 Table of Contents EXECUTIVE SUMMARY... 3 INTRODUCTION... 3 VIRTUALIZING APACHE HADOOP... 4 INTRODUCTION TO VSPHERE TM... 4 USE CASES AND ADVANTAGES OF VIRTUALIZING HADOOP... 4 MYTHS ABOUT RUNNING
AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap BIG DATA? Volume 59% growth in annual WW information 1.2M Zetabytes (10 21 bytes) this
Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica
SAP and Hortonworks Reference Architecture Hortonworks. We Do Hadoop. June Page 1 2014 Hortonworks Inc. 2011 2014. All Rights Reserved A Modern Data Architecture With SAP DATA SYSTEMS APPLICATIO NS Statistical
ENABLING GLOBAL HADOOP WITH EMC ELASTIC CLOUD STORAGE Hadoop Storage-as-a-Service ABSTRACT This White Paper illustrates how EMC Elastic Cloud Storage (ECS ) can be used to streamline the Hadoop data analytics
The 3 questions to ask yourself about BIG DATA Do you have a big data problem? Companies looking to tackle big data problems are embarking on a journey that is full of hype, buzz, confusion, and misinformation.
BIG DATA TRENDS AND TECHNOLOGIES THE WORLD OF DATA IS CHANGING Cloud WHAT IS BIG DATA? Big data are datasets that grow so large that they become awkward to work with using onhand database management tools.
Hadoop Ecosystem Overview CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook Agenda Introduce Hadoop projects to prepare you for your group work Intimate detail will be provided in future
Dominik Wagenknecht Accenture Improving Mainframe Performance with Hadoop October 17, 2014 Organizers General Partner Top Media Partner Media Partner Supporters About me Dominik Wagenknecht Accenture Vienna
Big Data, Why All the Buzz? (Abridged) Anita Luthra, February 20, 2014 Defining Big Not Just Massive Data Big data refers to data sets whose size is beyond the ability of typical database software tools
SQL Server 2012 PDW Ryan Simpson Technical Solution Professional PDW Microsoft Microsoft SQL Server 2012 Parallel Data Warehouse Massively Parallel Processing Platform Delivers Big Data HDFS Delivers Scale
Big Data: Business Insight for Power and Utilities A Look at Big Data By now, most enterprises have encountered the term Big Data. What they encounter less is an understanding of what Big Data means for
How Transactional Analytics is Changing the Future of Business A look at the options, use cases, and anti-patterns Table of Contents Abstract... 3 Introduction... 3 Definition... 3 The Expanding Digitization
5 Big Data Use Cases to Understand Your Customer Journey CUSTOMER ANALYTICS EBOOK CUSTOMER JOURNEY Technology is radically transforming the customer journey. Today s customers are more empowered and connected
Integrating Hadoop Into Business Intelligence & Data Warehousing Philip Russom TDWI Research Director for Data Management, April 9 2013 TDWI would like to thank the following companies for sponsoring the
Extending the Enterprise Data Warehouse with Hadoop Robert Lancaster Nov 7, 2012 Who I Am Robert Lancaster Solutions Architect, Hotel Supply Team firstname.lastname@example.org @rob1lancaster Organizer of Chicago
For: Enterprise Architecture Professionals Market Overview: Big Data Integration by Noel Yuhanna, December 5, 2014 Key Takeaways Big Data Creates New Data Challenges Today, most big data deployments are
Big Data for the Rest of Us Technical White Paper Treasure Data - Big Data for the Rest of Us 1 Introduction The importance of data warehousing and analytics has increased as companies seek to gain competitive
The Principles of the Business Data Lake The Business Data Lake Culture eats Strategy for Breakfast, so said Peter Drucker, elegantly making the point that the hardest thing to change in any organization
BEYOND BI: Big Data Analytic Use Cases Big Data Analytics Use Cases This white paper discusses the types and characteristics of big data analytics use cases, how they differ from traditional business intelligence
Forecast of Big Data Trends Assoc. Prof. Dr. Thanachart Numnonda Executive Director IMC Institute 3 September 2014 Big Data transforms Business 2 Data created every minute Source http://mashable.com/2012/06/22/data-created-every-minute/
SOLVING REAL AND BIG (DATA) PROBLEMS USING HADOOP Eva Andreasson Cloudera Most FAQ: Super-Quick Overview! The Apache Hadoop Ecosystem a Zoo! Oozie ZooKeeper Hue Impala Solr Hive Pig Mahout HBase MapReduce
A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani Technical Architect - Big Data Syntel Agenda Welcome to the Zoo! Evolution Timeline Traditional BI/DW Architecture Where Hadoop Fits In 2 Welcome to
Ganzheitliches Datenmanagement für Hadoop Michael Kohs, Senior Sales Consultant @mikchaos The Problem with Big Data Projects in 2016 Relational, Mainframe Documents and Emails Data Modeler Data Scientist
Big Use Cases To Start Today Paul Scholey Sales Director, EMEA 1 Exabytes of We all know the amount of data in the world is growing exponentially 40000 30000 YOU ARE HERE 20000 FROM 2010 TO 2015 77% of
Ensure PCI DSS compliance for your Hadoop environment A Hortonworks White Paper October 2015 2 Contents Overview Why PCI matters to your business Building support for PCI compliance into your Hadoop environment