Consistent, Reusable Analytics for Big Data: The Hallmark of Analytic Applications

Transcription

1 I D C T E C H N O L O G Y S P O T L I G H T Consistent, Reusable Analytics for Big Data: The Hallmark of Analytic Applications May 2015 Adapted from IDC FutureScape: Worldwide Big Data and Analytics 2015 Predictions by Dan Vesset, Carl W. Olofson, David Schubmehl, et al., IDC # Sponsored by Teradata This paper examines the value of big data applications as a means to ensure pervasive availability of big data analytics in the organization. Providers of big data tools have focused traditionally on data scientists, but using big data to support decision making is not the exclusive domain of this group. Managers, business analysts, and line-of-business (LOB) staff across industries and business processes stand to benefit individually and help their organizations be more competitive if they also have access to appropriate big data solutions. Big data analytic applications have the opportunity to make this happen. The Value of Big Data As the evidence mounts for the competitive edge demonstrated by organizations that rely on data-driven decision making, big data and analytics (BDA) have become top agenda items for a growing number of executives and organizations. At the same time, hype about big data technology capabilities and inflated promises of outcomes have subsided as early experimentation with solutions for managing and analyzing large volumes of diverse and fast-moving data has given way to pragmatic deployments. Yet, many organizations continue to struggle with technology, processes, and human resources needed to harness the power of big data for improving data-driven decision making. With the opportunity to unlock the value of big data to accelerate innovation, drive optimization, and improve compliance comes the need to demonstrate value, navigate expanding technology alternatives, recreate business processes, and ensure the availability of appropriately skilled staff. The ability to manage and analyze big data and derive value from these activities, as well as the ability to measure and improve these capabilities, will increasingly define an organization's ability to compete or service its constituents. New Expectations One of the primary drivers of the BDA market has been the new expectation for information access and analysis among technology end users not just data scientists but all users who interact with data. Popular search engines and round-the-clock focus on data scientists have set these expectations. First, many of us in the corporate world want and demand access to information that mirrors the ease with which we're able to find relevant information in our private lives. Second, the relentless message of ad hoc data access and analysis by a few analysts has awakened the curiosity of the broad majority of knowledge workers who want to be able to do the same at their level of expertise. Few are satisfied with the analytics functionality and data they are getting today. IDC research shows that on average only about 10% of end users across roles believe that the BDA solutions provided to them meet their decision support requirements to the fullest extent needed. This satisfaction rate is lowest (6%) for customer-facing employees and LOB managers (8%). IDC 1916

2 Employees in these roles are not expected to build their own predictive models or develop their own analytic applications. These employees "simply" want on-demand access to relevant data provided to them in the context of their business process and with "math" and user interfaces that are relevant and easy to use. For a marketing manager or analyst, this may mean an application that can answer questions about the effectiveness of a recently completed campaign or predict the likelihood of a positive response from customers to a change in a campaign. To the head of ecommerce, this may mean an application that answers questions about visitor or customer paths, sentiment, and shopping cart behavior. Within an industry, and across business processes in that industry, the roles and the questions they would like answered differ, and there is an opportunity to prebuild analytic applications to enable self-service information access and analysis and deliver faster time to value. The demand for such analytic applications is clearly there. In a study of over 1,200 managers and executives in 2013, IDC researchers found that business funds 61% of technology projects: 40% of projects are exclusively funded and executed by business, and 21% of projects are funded by business but executed jointly with IT. The reality is that LOB executives and managers are hungry for information and willing and able to spend to provide themselves and their staff with prebuilt analytic functionality that doesn't require deep expertise in application development or an advanced degree in statistics. So far, in the evolution of the big data market, the data scientist has been the hero. There is clear value to this role that combines statistics, computer science, and business domain expertise. However, data scientist is one of many roles that affects and is affected by an organization's BDA capabilities. Figure 1 shows a list of BDA-related roles, the shortage of which most hinders the success of BDA projects. Note that the shortage of skills to develop business intelligence (BI) and analytic applications ranks almost as high as the shortage of advanced and predictive analytics skills. FIGURE 1 Skill Shortages in Big Data and Analytics Q. Are your organization's BDA initiatives being hindered by a shortage of the following skills? Analytic skills, such as predictive model development, statistical analysis, and data or text mining Business intelligence technology development skills to build dashboards, reports, and analytic applications Business analysis skills such as multi-dimensional analysis, visual discovery, and spreadsheet analysis IT skills for providing the needed software infrastructure for data acquisition, management, integration, data quality, and data security IT skills for providing the needed hardware infrastructure Source: IDC, % 10% 20% 30% 40% 50% 60% 70% % of respondents IDC

3 To ensure that the expectations of all users are met, organizations need to provide appropriate data access and analysis technology to all knowledge workers. This does not mean deploying statistical analysis tools for every analyst and business decision maker, nor does it mean providing all LOB users with business intelligence tools regardless of how easy they are to use. Instead, there is a mounting need to provide this broader audience of "non-analysts," which has a growing appetite for data and analytically driven insights, with domain- and business process specific applications. These analytic applications represent a means to ensure more pervasive availability of BDA functionality throughout the organization. It's clear that not everyone can or should be a full-time analyst, let alone a data scientist. Analytic applications can incorporate the necessary data access, management, and analysis functionality delivered via industry- and/or business process specific applications that mask the complexity of, for example, predictive model development from users who appreciate its power but lack the expertise to develop, train, and deploy such models. Business Analytics Technology Trends IDC views big data as part of the broader business analytics technology market. Figure 2 depicts the segments of this $35+ billion global software market, which includes both tools and applications. Tools include databases, data integration, query, reporting, and advanced analytics software, and applications include prepackaged apps that incorporate the functionality provided by various tools but also have the following features: Business process support. Software that structures and automates a group of tasks pertaining to the review and optimization of business operations (i.e., control) or the discovery and development of new business (i.e., opportunity) Separation of function. Can function independently of an organization's core transactional applications, yet can be dependent on such applications for data and may send results back to these applications Time-oriented, integrated data from multiple sources. Extracts, transforms, and integrates data from multiple sources (internal or external to the business) supporting a time-based dimension for analysis of past and future trends or accesses such a data store. Some examples of analytic applications by function are: Pricing optimization in retail Predictive maintenance in asset-intensive industries Fraud detection and prevention in banking Churn detection and prevention in telecommunications Logistics optimization in transportation Consumer sentiment analysis in media and entertainment Market basket analysis in retail Some applications can be highly specialized to a subprocess within an industry, while others are more generalized and could be applicable across industries IDC 3

4 FIGURE 2 IDC's Business Analytics Software Market Segments Performance Management and Analytic Applications Financial Performance & Strategy Management (consolidation, profitability, budgeting, planning, strategy mgmt) CRM Analytic Applications (marketing, sales, customer service, call center, Web site, price optimization) Workforce Analytics IT Operations Analytics Supply Chain Analytic Applications (procurement, asset mgmt, logistics, inventory, manufacturing) Services Operations Analytics (healthcare, education, financial services, government, communications & media) Product Planning (demand, supply, and production planning) Business Intelligence & Analytics Tools Query, Reporting, & Multidimensional Analysis (production reporting, query, OLAP, visual discovery) Advanced & Predictive Analytics (modeling tools and servers for statistics and data mining) Content Analytics (text analysis, search, rich media analysis, cognitive systems) Spatial Information Analytics Continuous Real-Time Analytics Software Information Management Software Relational Data Warehouse Management Non-Schematic Data Management Software (NoSQL, Non-Relational, Hadoop) Data Integration (ETL, Data Quality, Data Governance) Source: IDC, 2015 Analytic applications have existed for many years, and IDC has tracked the market adoption of these apps since the mid-1990s. These applications developed over time as the functionality and performance of the underlying technology components and tools solidified and as opportunity arose to package experience from lessons learned from years of deployments into applications. That meant developing functionality that incorporated workflows common in a particular industry and business process as well as metrics or KPIs commonly associated with decisions being made in specific domains. Over time these applications have expanded to include support for collaboration, prescriptive advice to users to walk them through the process of data analysis, and functionality to push results of analysis to downstream systems. Although some transactional applications over the years have incorporated more query and reporting functionality to support business intelligence and analytics needs, only 14% of end users in a recent IDC survey said that their organization's enterprise applications (e.g., ERP, CRM, SCM) extensively incorporate analytics, BI, or decision support functionality. The demand for analytic applications continues, and it has begun to emerge in the big data segment of the market where large, highly varied, and fast-moving data sets and streams are being analyzed, often with predictive analytics models using path, pattern, text, graph analysis, and machine learning techniques IDC

5 The Next Phase of the Big Data Analytic Applications Market Many of the challenges and inhibitors to successful BDA solution deployments have been exacerbated in the big data segment of the market. In the past it was possible to get by with a "basic" database and spreadsheet skills for data analysis, but today multiterabyte- and even petabyte-sized data sets, the combination of structured and unstructured data from internal and external sources, and often the speed at which data is being generated from the increasingly instrumented world are putting new pressures on all organizations. As a sign of the next phase of maturity, the big data market is ripe for the introduction of analytic applications. These solutions can have a number of benefits for organizations and their LOB and data science employees. Benefits Organizations looking to deploy big data analytic applications shouldn't view them as replacements for various big data management and analysis tools and components. There will continue to be a need for custom-developed solutions and the free-form access required by data scientists to explore raw data sets. However, organizations also can derive the following key benefits from big data analytic applications: Faster time to market and time to insight. Process, role, and industry specificity allows organizations to configure and deploy prepackaged applications faster than a similar custom development project would require. IT can engage with analytics and business groups to set up, manage, and organize applications. Business users can take advantage of "out of the box" data visualization and analytic methods that are preconfigured for them to ensure contextual relevance. Better utilization of scare resources. Data scientists can focus on the most complex cases requiring custom development and custom analytics. At the same time, data scientists can capture program logic in apps allowing collaboration, sharing, and reuse with LOB colleagues. Reusability. Big data analytic apps make it possible for data scientists and analysts to focus on new analytics initiatives without having to repeat projects; IT personnel no longer have to maintain a large variety of custom-built, single-purpose analytics solutions. Considering Teradata One of the vendors that has entered the big data applications market is Teradata. The company's big data analytics and discovery solution, Aster, has four key components: Teradata Aster Discovery Platform provides a high-performance analytic environment that allows analysts to combine SQL, MapReduce, Graph, and R within a single query for big data analytics and discovery. Aster provides prebuilt analytics like path, pattern, text, graph, and machine learning to tease out insights from multistructured data. Teradata Aster AppCenter (AppCenter) is a Web-based solution with a point-and-click interface for building, deploying, sharing, and consuming interactive, big data applications. It consists of a Web-based interface for configuring, parameterizing, organizing, and finding apps. The apps are written in SQL or Java and have access to over 100 prebuilt algorithms around path, text, graph, and machine learning for big data analysis. AppCenter enables business users to visualize and share results through interactive charts and graphs. The solution also includes an SDK and common application services like user authorization and data access management as well as app execution, scheduling, and logging services for streamlined application building and deployment. AppCenter supports external tool integration through RESTful API, JSON objects for visualization via third-party tools, and URLs for embedding application results into Web-based interfaces IDC 5

6 Big Data Apps from Teradata are industry-focused, analytic apps that capture best practices for analytics in customizable, prebuilt templates that answer specific business questions and are available for a number of industries. The templates address a wide range and a growing number of use cases across industries all at intersections of specific business processes within a given industry. Examples of these apps include analysis of fraud networks in consumer financial services, companion matcher in gaming, drug prescription affinity analysis in healthcare, shopping cart abandonment analysis in retail, paths to churn in telecommunications, and customer review and sentiment analysis in travel and hospitality. Teradata Professional Services works with client organizations to accelerate implementation of big data apps by configuring the program logic, schema, visualization, security, and access interface captured in the templates to meet specific business requirements. A typical flow when engaging with big data apps from Teradata follows these steps: Configure the app. Teradata Professional Services will work with your organization to configure the big data apps of choice to your data and specific requirements. Run the app. A variety of users can run the app or schedule the app for a future run. Prior to running the app, users can add a report title, change parameter values, select visual output choices, and create custom filters and more. Discover insights. Users can interact with results through a visual interface that includes charts and graphs and can start operationalizing the insights or have an iterative analysis process to uncover more or new insights by changing the parameters and rerunning the apps. Challenges and Opportunities for Teradata Like all vendors in the broader BDA market, Teradata faces competition. It must convince a broad audience of business leaders about the value of big data applications and ensure that it facilitates a collaborative environment between LOB, analytics, and IT groups. This is not a trivial task. It will require a combination of technology and business consulting skills. Teradata is in a strong position to address this market challenge based on its extensive professional services organization. There is also the challenge of addressing a very wide range of discrete requirements across functions and industries. No single IT vendor can provide all apps to everyone. Teradata's approach of providing an AppCenter along a portfolio of some specific applications is likely to attract partners that can help build additional applications in their own areas of expertise. This approach should enable the company to provide the most relevant big data applications to the broadest possible audience. As always, IDC recommends that technology buyers evaluate available choices based on a number of functionality, performance, financial, and risk variables. Conclusion The growing focus on BDA solutions as a basis for competitive advantage is both an opportunity and a challenge for most organizations. The promise of better and faster data-driven decision making has pushed BDA technology to the top of executive agendas. In this environment, not only access to information but also the ability to analyze and act upon information in a timely manner creates competitive advantage in the marketplace, enables sustainable management of communities and natural resources, and promotes appropriate delivery of social, healthcare, and educational services IDC

7 Recommendations Identify specific business issues to address with big data. The first task, as with any technology, is to identify business challenges and opportunities where new data and new combinations of data as well as new analytics, visualizations, and metrics can have a meaningful impact on outcomes. Big data projects are no longer simply experimentations based on theoretical questions and excitement about new technologies. The market has entered a new, pragmatic phase of big data technology adoption and deployment where value is being derived from well-defined and well-scoped projects supported by appropriate technology. Decide when to build versus buy versus configure. There will always be big data uses that require custom development. Data scientists have a key role to play, and their value is critical to the organization. However, not every business process needs to be supported by one-off, custom analysis and development efforts. Many operational and tactical decisions are repeatable and well suited for operationalization in the form of analytic applications. Look for well-established decision processes where data sources and workflow are defined and understood by all the stakeholders. These applications will still require parameterized inputs from end users and periodic review of results by analyst and data scientists. However, the repeatable nature of the decision being supported will allow for technology deployment and efficiencies as well as decision-making consistency that can yield positive outputs for the organization. Consider big data applications as an emerging viable option. Analytic applications have existed for many years, but the emergence of big data disrupted the market for existing applications. With the high volume, wide variety, and high velocity of data came new technology components that necessitated a new cycle of experimentation, proof of concepts, and learning. Now that the initial phase of these activities is over, technology vendors will be moving to produce and support prepackaged big data applications that are productive and operationalize the lessons learned from several years of discrete big data tool deployments and customization. A B O U T T H I S P U B L I C A T I ON This publication was produced by IDC Custom Solutions. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Custom Solutions makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. C O P Y R I G H T A N D R E S T R I C T I O N S Any IDC information or reference to IDC that is to be used in advertising, prior written approval from IDC. For permission requests, contact the IDC Custom Solutions information line at or gms@idc.com. Translation and/or localization of this document requires an additional license from IDC. For more information on IDC, visit For more information on IDC Custom Solutions, visit Global Headquarters: 5 Speen Street Framingham, MA USA P F IDC 7