Analytics in the Cloud

Transcription

1 Analytics in the Cloud Five Components for Success An ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) White Paper Prepared for Teradata Corporation April 2014 IT & DATA MANAGEMENT RESEARCH, INDUSTRY ANALYSIS & CONSULTING

2 Table of Contents Analytics and the Cloud... 1 Teradata Cloud... 2 Amazon Redshift Data Warehouse... 3 Five Components for Successful Cloud-Based Analytics... 3 Analytic Performance... 4 Flexibility... 5 Advanced Technologies... 6 Expert Support... 6 Enterprise Ecosystems... 7 Cloud Analytic Use Cases... 7 Utilizing the Cloud to Report and Deliver Insight to Partners... 8 Cloud-Based Customer Segmentation... 8 Prepare for Success EMA Perspective... 11

3 Analytics and the Cloud Cloud solutions have arrived and are here to stay. Innovative companies are reviewing their data management strategies to identify where and how Cloud should play a role. These new programs are driven by an array of Cloud value propositions and technology advancements that enable companies to disrupt traditional data management paradigms in favor of new ways to create value. Analytics has followed this trend and is providing new opportunities to serve enterprise end users and projects. Cloud-based analytics are becoming commonplace as most industry sectors identify opportunities to augment or replace existing infrastructure with this cutting-edge technology. This adoption trend parallels ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) research 1 that identifies growing trends toward a more hybrid data management landscape focused on leveraging purpose-built platforms, such as Cloud, to align data, applications, and workloads for performance and cost advantage. Sixty percent of respondents to recent EMA research indicate they are using two to three different platforms to execute sophisticated workloads and Cloud often plays a critical role in this mix. As strategies shift toward Cloud platforms, other business and technical drivers are enhancing the opportunities presented by Cloud analytics. Faster Time to Value: Traditional on-premises projects can often take months to launch. Provisioning and deploying hardware and software is a time consuming process. Procurement and implementation must be prioritized into IT schedules, and there is usually a wait for IT asset deployment. Cloud analytic solutions cut months into weeks and in some cases days. Economics: The capital expenditure involved in traditional analytic projects can be a significant roadblock both financially and from a time-to-implementation perspective. Cloud analytics eliminates the need to purchase expensive software and hardware and offers a subscription-based pricing model that helps to control overall project costs. Many Cloud platforms enable users to grow and contract projects, thus providing flexibility in costs that impact operational expenses. Enterprise Ready: The technology available to Cloud analytic solutions has evolved quickly and proven to be enterprise ready. Leading vendors have created platforms that deliver proof ofconcept projects, development environments, and production environments. These environments are ready to take on the workloads of innovative companies who need to address challenges arising from traditional on premises solutions. Agility and Flexibility: Cloud solutions are addressing the needs of analytic clients who require a more flexible and agile architecture. Fast provisioning, elastic computing power, and data storage are providing a level of agility not often found in traditional solutions. Technology Advancements: Cloud technology has advanced to address sophisticated needs of enterprise customers. Leading solution providers are leveraging technology advancements to deliver advanced capabilities that early adopters didn t initially require. Speed, stability, and sophisticated security have led these solutions to become enterprise viable, thus allowing companies to make Cloud analytics a key component to their data strategy. Cloud technology has advanced to address sophisticated needs of enterprise customers. In this EMA paper we will examine two Cloud solutions for data warehousing and analytics Teradata Cloud and Amazon Redshift. First, let s briefly review each vendor s Cloud offer. 1 Big Data: Operationalizing the Buzz, Enterprise Management Associates and 9Sight Consulting, November 2013, 1 Page 1

4 Teradata Cloud The Teradata Cloud solution is delivered in three as a service models : Data Warehouse as a Service (Teradata Database) Discovery as a Service (Teradata Aster Database) 2 Data Management as a Service (open source Apache Hadoop ) 3 Within each as a service model, there are typically three components (see Figure 1): Cloud Foundation Cloud Foundation is the base offer for Teradata Cloud and delivers hardware/ software monitoring and management along with administration, security, resource provisioning, daily backup, maintenance, and customer on-boarding services. Enhanced Services These optional services focus on delivery of advanced features and functions specific to security (such as column level encryption), the business intelligence layer, data integration, and data quality. Consulting Services Teradata provides consulting services to assist clients with best practices in database design, data migration, database management, advanced analytics, and application integration. Figure 1 Teradata Cloud as a service models Teradata Cloud offers operational expertise in Teradata Database, Teradata Aster Database, and open source Hadoop. They apply deep industry knowledge drawing upon years of analytics expertise. Teradata also delivers industry-specific accelerator kits that include data load scripts; high-level data models and Business Intelligence (BI) report templates coupled with professional services to help clients get started in the Cloud much faster. Teradata Cloud offers operational expertise in Teradata Database, Teradata Aster Database, and open source Hadoop. 2 Generally Available mid Already available for on-premise installation. Available for Cloud in Page 2

5 Amazon Redshift Data Warehouse Amazon Redshift is a managed Cloud data warehouse solution. Redshift leverages the Amazon Cloud system to provide a global infrastructure that can be deployed in just minutes. Redshift boasts extremely competitive and flexible pricing models that enable companies to start their projects with no commitments or upfront costs. The platform can scale to one petabyte or more allowing subscribers to realize an advertised price of $1,000 per terabyte per year for the service. Figure 2 Amazon Redshift Architecture Redshift partners with leading solution providers to extend function and value of their solution to include business intelligence, data integration, data quality, and data visualization features and functions. Redshift is compliant with SOC1, SOC2, SOC3, and PCI DSS level requirements. Five Components for Successful Cloud-Based Analytics While Teradata Cloud and Amazon Redshift each excel in various areas, there are critical considerations for successful cloud-based analytics: Analytic Performance Flexibility Advanced Technologies Expert Support Enterprise Ecosystem Each consideration will be reviewed in detail. 3 Page 3

6 Analytic Performance Analytic performance is a critical factor in deciding on an appropriate Cloud solution. Performance criteria can take on many attributes including data acquisition or the ability to load data into an analytic environment. Next is the amount of data that an environment can support overall along with the speed at which the answers to business questions can be generated (via queries). Finally, there is the consideration of how many people in the organization can ask business questions at any one point in time and receive result sets in a reasonable timeframe. Determining how easily and quickly information can move from a source system to the Cloud-based analytics environment is critical in supporting various analytic use cases. Slow or cumbersome data acquisition will impact how quickly decisions can be made within an organization. Laborious data preparation and multi-step data loading hurdles can slow productivity as well, and add additional ongoing operational costs to a project. To support sophisticated workloads, avoid adding complexity and manual processes to this critical function. Another performance constraint examines how much information can be stored in a Cloud-based analytic environment. This dictates the amount of history associated with business questions. Relatively small or space constrained platforms will limit the length of comparison in business questions. Analytic implementations with higher overall storage capabilities allow an organization to look back over a year or longer to identify trends or make business comparisons. Systems unable to handle higher data scaling can and will cause restrictions on fast growing projects. Finally, Cloud-based platforms need to support not just a single user asking business questions, but allow multiple users to query the system at the same time. Insights must be processed and returned in a timely fashion no matter how many people are querying the system. Long queues and delayed insight will restrict adoption and reduce or eliminate ROI. Analytic performance is a critical factor in deciding on an appropriate Cloud solution. Cloud-based platforms need to support not just a single user asking business questions, but allow multiple users to query the system at the same time. Teradata Cloud and Amazon Redshift offer various data integration options. In the case of Redshift, customers will need to use a multi-step approach to loading data into the Cloud solution. After preparing data and splitting it into the appropriate number of files the data needs to be moved to Amazon s Simple Storage Service (S3) platform. From S3, a copy command is executed to move data into the Redshift environment. This step is executed in a Massively Parallel Processing (MPP) fashion helping this part of the process to move quickly. Teradata Cloud utilizes a more direct data integration procedure requiring less manual data preparation, achieving faster data load times than Amazon Redshift. Both solutions are designed to engage large quantities of data and have the solid capability to scale. As these Cloud-based analytic environments grow, Teradata demonstrates the ability to maintain speed with scale at a better rate than the Amazon Redshift environment. The final performance trait is centric to the issue of query concurrency, or the number of queries the environment can process at a single time. Redshift offers the ability to maintain 500 database connects but is limited in its number of concurrent queries. This limitation will fall short for use cases that are serving large enterprise user communities. Teradata Cloud is designed to extend well beyond this limitation on concurrent queries. 4 Page 4

7 Flexibility Workload flexibility is a critical fork in the road when considering a Cloud-based analytic solution. Most platforms need to respond to a variety of analytic workloads. Reporting, iterative OLAP, ad-hoc and data mining or advanced analytics are common to these environments. Defined business questions regarding the financial or operational status of a line of business, business unit or organization (reporting). These questions are generally well known and can be optimized based on the standard answers. Dynamic and changing questions about an organization s activities are (ad hoc). Often these questions require links between disparate datasets and are not optimized in advance. Linked questions require one set of data and then feed into subsequent queries (iterative OLAP). These queries answer questions of the what if variety where one question s answer leads to the next question. Advanced analytic questions often relate to use and support of predictive/data mining. This can generally require the use of outside processing models, often addressed in database. Supporting a variety of workloads is the hallmark of an agile and flexible platform. Database orientation is a feature used to enhance performance and flexibility and serves the workloads above in different ways. Traditional row-based strategies enable ease of inserts and updates to records but can have a disadvantage to properties of columnar layouts when queries scan entire tables for information. Some platforms will have a mix of columnar and row-based technologies to meet the demands of the analytical workloads above. Supporting a variety of workloads is the hallmark of an agile and flexible platform. Choice of analytics can also be demonstrated in variety across a Cloud offering and how it s designed to execute more advanced analytic approaches. Big data strategies that require Hadoop infrastructure and discovery analytics that power a deeper capability to explore and analyze data are critical to many companies looking to optimize their existing analytic capabilities. The size of a Cloud analytic environment will grow over time. It s important that the platform is able to scale to meet these needs and to do so economically and with little system interruption. And most Cloud analytic solutions focus on delivering a pricing format that enables growth in users and environment size over the life of a project. These flexibility issues can be as important as the technical specifications of a solution. Amazon Redshift delivers its solution in a columnar format and has several advantages over certain row-based workloads. Teradata Cloud offers a more flexible hybrid environment allowing clients to take advantage of row-based and columnar data organization in the same solution. Teradata s Cloud solution integrates with the Teradata Aster Database, as well as Hadoop, to extend its capabilities to include big data analytics, with functions focused on easier fraud detection, machine data analysis, text analysis, and social data analysis. Amazon doesn t offer a discovery platform at this time and its Cloud Hadoop solutions are not as integrated to Redshift when comparing the two solutions. Amazon Redshift and Teradata Cloud both offer a resizing strategy that allows a larger shadow environment to be spun up in advance allowing the user to switch over without significant system down time. 5 Page 5

8 Advanced Technologies Successful analytic projects often evolve beyond their initial scope. After a time, most are holding greater quantities of data than initially planned, as more users have adopted the platform and the initial set of feature requirements are often found to be somewhat short-sighted. This is why it s critical to plan beyond initial scope and push project requirements forward even though at the time it may not seem necessary. Successful analytic projects often evolve beyond their initial scope. In the section above, we explored various types of data loads and query work that might be executed on a Cloud analytic platform. As a data-driven project matures so will the need for more advanced features and functions. This is especially true as users demand insights that go beyond traditional system functionality. The ability to extend and integrate a Cloud analytic environment with innovative functions such as discovery-based analytics creates an immediate need for a platform capable of executing this style of work. And orchestrating a data warehouse along with a Hadoop solution in the Cloud presents exciting opportunities for advanced insights. Ensuring organizations have the option to expand into these new areas of execution can help a project stay on track, grow and deliver unexpected value. Advanced database features and functions parallel this topic. Many Cloud solutions are somewhat new or immature in their development and are not always designed to work with certain data types or able to execute some advanced database functions. To keep pace with the innovation driven by users, it s important to explore existing requirements as well as future needs. The Teradata Cloud offers Discovery as a Service Teradata Aster solution as well as its own Data Management as a Service that connects open source Apache Hadoop to the environment. Redshift can be integrated with other Amazon data solutions, but at this time a discovery platform is not offered. Amazon does offer Hadoop in the Cloud but struggles to orchestrate the environment in an end-to-end fashion. Redshift is presently designed to execute standard database functions that support reporting workloads. The platform is based on PostgreSQL and has some limitations with regard to supported data types and functions. At present Redshift is unable to work with Temporal (time) data, JSON, XML, and geometric data types that support geographic analysis. At the time of writing this paper, Redshift also did not have a Hadoop connector. Teradata Cloud delivers functionality on each of these. Expert Support The provisioning of a database infrastructure is a critical aspect of implementing an analytic solution; however, it is not the only aspect of solutions implementation and design. To support important business questions, it is necessary to have information in the proper organizational format. This allows a Cloud analytics platform to effectively deliver accurate and relevant information to business users. Building and designing an architecture is not a trivial project. It may require advanced database management skill sets and experience with data modeling, data integration and security, just to name a few. Many companies, especially enterprise-level firms have full-time employees who specialize in these tasks and can contribute to the process. Companies that move to the Cloud are often challenged by IT skill issues and can t always get the new project support that matches the required speed of implementation on Cloud projects. Reliance on the Cloud solution provider for professional services, training and implementation assistance is paramount and the differences found between vendors will act as a guide-post to how an organization should plan its strategy. 6 Page 6

9 If there is a robust definition of these data management concepts within an organization, a Cloudbased analytics platform can be a relatively fast implementation because it would utilize pre-existing definitions and data locations and then migrate them to the Cloud system. However, if a company doesn t have those definitions and data locations, the project may require expertise to model the data. One option is to utilize pre-packaged industry solutions that allow for an organization that doesn t have the deep domain expertise of a new product line or a new business to migrate those definitions to a Cloud-based analytic environment just as easily as it would its own custom-built definitions. In the case of Amazon Redshift and Teradata Cloud, there are vast differences in how these companies support data migration and data management processes. Amazon Redshift offers support by phone and can help companies engage its partner network to attain the expertise required to design the platform and organize the data to support the needs of its users. Teradata delivers its own certified professional services organization to support design and deployment. Additionally, Teradata s professional services focuses on helping clients leverage the finished solution for optimal business value including mentoring and training on analytic practices, and daily data management and operations support. Lastly, Teradata can deliver industry accelerator kits that enable users to move forward with pre-defined load scripts, data models and BI reports that match business requirements of companies within specific industry verticals. On the service front, both firms provide multiple tiers of customer service ranging from phone support on up to onsite customer visits. Teradata offers a dedicated account manager relationship that often is sought by enterprise-level firms. Enterprise Ecosystems All analytic platforms need the ability to work within a wider ecosystem. Having the business questions/insights locked into a single platform limits the value of those insights. Also, coordination with both downstream data consumers (be they actual users or consuming platforms) and with administrative facilities such as archive, operational management, and monitoring makes a Cloud analytic solution more effective. In addition, leveraging the capabilities of advanced analytic platforms or discovery platforms can enable sophisticated workloads and projects. All analytic platforms need the ability to work within a wider ecosystem. Most, if not all, solution providers offer extensive partner networks to extend capabilities and function. These partners can add value to a project but at the same time can add complexity and management overhead. In comparing Teradata and Redshift, both firms invest in vibrant partner networks that include BI, ETL, and other common functions. Teradata orchestrates the management and performance of BI and ETL solutions in the Cloud on behalf of its clients to provide a more seamless experience. Redshift customers will need to take on the burden of managing these additional solutions, as they are required by the project. Cloud Analytic Use Cases Companies from all industry sectors are investigating Cloud analytics to drive innovation. Use cases vary widely as does the technology delivered by Cloud vendors. It s critical to choose technology that can deliver on the specific performance needs of a project. Many early stage programs are focused on duplicating or expanding existing reporting functionality of a company. As discussed above, CAPEX and OPEX can be positively affected by adopting Cloud to replace or augment existing traditional systems. 7 Page 7

10 Utilizing the Cloud to Report and Deliver Insight to Partners Progressive companies work diligently to better communicate with their partners to influence and streamline performance. A common Cloud analytics use case is centered on reporting between companies and suppliers. Partner information portals deliver valuable data enabling partners to react to trends, inventory levels, market insights, and service issues. Implementing a reporting platform that s available through a Cloud-based analytic solution allows companies to leverage a system they don t need to purchase outright but can lease for the duration of the project. Most of the workloads built into these reporting-centric solutions are predefined and narrowly delivered to each specific partner group. Partner portals deliver insights on their specific set of supplied products or services as well as aggregated competitive performance information that allows partners to gauge their success against other products services in the same sectors. Figure 3 - Cloud Reporting Architecture Cloud-based analytic solutions designed to take on these types of reporting workloads are generally architected in a direct and simple manner, accessing data from sources behind corporate firewalls or other Cloud-based applications to supply the appropriate information to the Cloud platform. Standard SQL-based reporting tools can then be directed toward the data to supply the interface necessary to serve the retailers partner ecosystem. Reporting workloads are limited in scope so they don t present exaggerated challenges to the Cloud platform and are a great fit for many Cloud service providers including both Teradata and Amazon Redshift. Cloud-Based Customer Segmentation Less sophisticated Cloud projects can be temporary in nature or limited in scope from an enterprise standpoint. The example above explores a reporting centric workload, serving a limited number of users in a highly focused manner. Many companies are making a longer-term and more mission-critical bet on Cloud analytics, thus making the Cloud a central component of their data management strategy. 8 Page 8

11 Companies that service extremely large and diverse customer bases have found traditional technologies and architectures limited in their capacity to meet their growing challenges. Understanding and segmenting extremely large sets of customer data is key to their ability to make product recommendations and next-best product offers to their best customers in near real-time environments. This type of data-driven workload goes well beyond the simple reporting use case listed above to leverage multiple platform functionality within the organization. Cloud analytics can play a key role in these sophisticated ecosystems and enable innovation while supporting mission-critical work. Figure 4 Complex Cloud Architecture To deliver targeted, next-best product and service offerings, organizations rely on an array of solutions to work in unison. These platforms are under extreme pressure to perform and handle the data and query volumes necessary to execute. These projects often leverage a diverse set of data in order to deliver the necessary business insights. When comparing and contrasting these use case examples, several themes emerge that differentiate both workload and platform requirements. Reporting workloads are more easily delivered in the Cloud, often accessed with lower concurrency of queries and with less need to address disparate data. Both Teradata and Redshift would be suitable for these types of reporting. However, the next best offer scenario calls upon a wider variety of data and requires a greater level of orchestration and execution from multiple platforms and technologies. This type of analytic could be better driven from Teradata Cloud. 9 Page 9

12 Prepare for Success The use case examples described in this EMA white paper draw an interesting line between workloads and Cloud platform capabilities. As illustrated, issues of data volume, concurrency, and complexity all play a role in understanding a specific platform s ability to function and grow with changing business conditions. Cloud-based projects often suffer an inability to shift or morph as new demands are applied to the infrastructure. It s critical to investigate the growth potential of a Cloud solution to ensure it will grow with a company s needs. Quite often, Cloud analytic projects start as a proof-of-concept and are pressed into production quickly. These early-stage projects can grow exponentially over time and expose weaknesses in the platform, eliminating much of the project s upfront Return on Investment (ROI). So be careful to look before you leap. The Cloud platform selected today should be agile and flexible enough to meet tomorrow s needs. When investigating Cloud-based analytic solutions, pay special attention to the platform s core capabilities today as well as the future roadmap (if available). Determine if the solution can meet the expected size, data volumes, and workloads of the user community. Determine if the platform is geared for reporting with limited numbers of concurrent queries, or if it s designed to support larger enterprise workloads that will likely better match future needs. Pay special attention to the platform s core capabilities today as well as the future roadmap. All technology solutions are driven by common capabilities, especially those centric to analytics and data management. To execute against a long-term strategy, invest time in better understanding the following criteria: 1. Licensing Models While the financial side of Cloud analytics has many positives, it s still important to research the way an organization will license a vendor s technology to ensure it matches long-term strategy. 2. Open Architecture Ensure the solution supplies robust APIs and connectivity to other Cloud and traditional on-premises systems to avoid creating an isolated analytics instance. 3. Industry Knowledge Generic solutions can slow time to value creating longer implementation cycles and put greater demands on the project team. Look to vendors who can supply strong industry knowledge to jumpstart new projects and set them on the proper course for success. 4. Solution Ecosystem Understand the added value that mature vendors can bring to projects. This may include capabilities to integrate business intelligence, data integration, and other technologies necessary to derive expanded value from Cloud analytic investments. These criteria will assist an organization in better planning for the long-term success of a Cloud analytic program. 10 Page 10

13 EMA Perspective Cloud analytic solutions have become a first-class citizen of an innovative data management strategy. But not all Cloud solutions are created equal, and it s becoming more important as workloads and sophistication of applications evolve, to select a platform for the correct use case. Performance, flexibility, advanced technologies, expert support, and enterprise ecosystems are all critical components to selecting the correct platform to meet the present and future needs for Cloud enterprise analytics. Each of the solutions discussed in this paper delivers value for its clients and has compelling capabilities. Cloud analytic solutions have become a first-class citizen of an innovative data management strategy. Amazon Redshift is ideally positioned to solve challenges for mid-sized companies or departmentallevel projects where reporting is the primary workload, concurrency is low, and data volumes are kept in check. Teradata Cloud s functionality coupled with its enhanced services and consulting capabilities can help to take an analytic strategy beyond entry-level programs, to projects that are mission critical and more highly integrated into enterprise data ecosystems. The wider selection of services enables customers to jumpstart their initiatives with professional guidance, industry quick-start templates and analytics experience. About Enterprise Management Associates, Inc. Founded in 1996, Enterprise Management Associates (EMA) is a leading industry analyst firm that provides deep insight across the full spectrum of IT and data management technologies. EMA analysts leverage a unique combination of practical experience, insight into industry best practices, and in-depth knowledge of current and planned vendor solutions to help its clients achieve their goals. Learn more about EMA research, analysis, and consulting services for enterprise line of business users, IT professionals and IT vendors at or blogs.enterprisemanagement.com. You can also follow EMA on Twitter, Facebook or LinkedIn. This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission of Enterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. EMA and Enterprise Management Associates are trademarks of Enterprise Management Associates, Inc. in the United States and other countries Enterprise Management Associates, Inc. All Rights Reserved. EMA, ENTERPRISE MANAGEMENT ASSOCIATES, and the mobius symbol are registered trademarks or common-law trademarks of Enterprise Management Associates, Inc. Corporate Headquarters: 1995 North 57th Court, Suite 120 Boulder, CO Phone: Fax: Teradata is a registered trademark of Teradata Corporation and/or its affiliates in the U.S. and worldwide. EB-8048 > Page 11