Business Usage Monitoring for Teradata

Transcription

1 Managing Big Analytic Data Business Usage Monitoring for Teradata Increasing Operational Efficiency and Reducing Data Management Costs How to Increase Operational Efficiency and Reduce Data Management Costs

2 Table of Contents 3 Executive Summary 4 Big Data getting bigger and analytic complexity exploding 5 Inadequate Legacy Monitoring Tools 6 Visibility Needed to Increase Efficiency and Reduce Costs 8 Business Usage Monitoring and Best Practices 11 Appfluent Visibility and Teradata Viewpoint 12 Conclusion 17 2

3 Executive Summary As technological advances have reshaped business, government and consumers, Business Intelligence (BI) applications and data warehouse deployments have grown from departmental to enterprise-wide in recent years. As a result the appetite for data is insatiable and analytic data volumes are growing exponentially with tens, hundreds of terabyte and petabyte scale data warehouse systems becoming the norm. With exploding data volumes and increasing analytic complexity, Information Technology (IT) managers are under siege to respond to business needs while reducing the costs associated with data delivery. Unfortunately, data managers, application database administrators, data architects and analytic application managers do not have the required instrumentation to gain visibility and understand what data is used, unused and more importantly, deciphering how data is being used to retain and optimize the most relevant assets. However to increase operational efficiency and reduce data management costs requires the collaboration of several functional groups within IT. Legacy database monitoring tools are primarily designed for system database administrators and do not provide the necessary instrumentation for senior data managers, data architects, application managers and data integration professionals. With adequate Business Usage monitoring enterprises can gain visibility into business activity and data usage to deliver increased operational efficiency, ensure scalability of both infrastructure and available IT resources and reduce data management costs. The key objectives of Business Usage monitoring are to: Justify costs, prioritize and invest resources based on business utilization Retain and optimize the most relevant data and processes Respond faster and ensure scalability and performance To gain tangible benefits a comprehensive solution for usage monitoring should be combined with best practices to analyze and take meaningful action. In addition, effective usage monitoring requires a solution that integrates with the BI, data warehouse and data integration stacks to provide a complete view into business activity and data usage. Appfluent Visibility for BI and Data Warehousing is a software solution that monitors how business units and departments use data so that IT organizations can improve operational efficiency, scalability, performance and control data delivery costs. 3

4 Big Data getting bigger, analytical complexity exploding Over the last several years, organizations have invested heavily in Business Intelligence (BI) applications and data warehousing to provide better access to enterprise data. The deployment of analytic applications has evolved from small departmental power users using disparate tools to widely accessed enterprise business intelligence applications. As business usage of analytic applications has exploded so has the appetite for more data. Big Analytic Data in data warehouses is getting bigger by the minute as the amount of raw data and number of source systems continues to burgeon. Data stored in data warehouses is now growing into the tens and hundreds of Terabytes very rapidly. A recent study conducted by the Aberdeen Group showed that large enterprises witnessed a 41% growth in data year over year from 2009 to 2010 (Source: Data Management for BI Aberdeen Group, December 2010). While the data growth is exploding, the same study also found that 50% of the organization expressed that a lot of the data is not accessed or underutilized by the business. In this exploding environment, Information Technology (IT) managers have the unenviable task of responding faster to the increasingly demanding business needs while reducing the costs associated with data delivery. Additionally, the pressure on IT organizations to control costs comes at a time when organizations are mandating expanded deployments of BI and data warehouse implementations. Compounding the problem is the fact that an enterprise deployment of BI and data warehousing requires complex interaction and collaboration between different functional groups before, during, and most importantly after the implementation. In addition, unlike off-the-shelf transactional applications, a BI and data warehouse deployment is constantly evolving. The number of users, query volume and query complexity changes with alarming irregularity as data marts and data warehouses grow in size and complexity. Unfortunately, most inefficiencies in BI and data warehousing environments arise from a lack of understanding of how applications and data are actually used across the organization. Data managers, application database administrators, data architects and analytical application managers are hampered by a lack of tools that provide visibility into understanding what data is used, what is unused and more importantly, understanding how data is being used. This information is critical in order to retain and optimize the most relevant data assets. 4

5 Inadequacy of Legacy Monitoring Tools To help address the issues noted above, most call upon legacy application and database monitoring tools or application audit logs. However, these solutions are simply not designed to provide the necessary visibility needed to understand business activity and data usage to help organizations manage data warehouses more efficiently. Legacy application monitoring tools are primarily designed for monitoring and tracing transactional applications. They provide application developers the ability to conduct load testing and transaction profiling necessary to design, test and deploy transactional applications in production. The legacy application monitoring tools are not designed to provide visibility into how the business uses analytical data. Legacy database monitoring tools are designed primarily for system database administrators and focus mainly on monitoring system metrics such as CPU, memory, buffer pools, and SQL explain plans to tune the databases. The legacy database monitoring tools do not integrate with analytical applications and therefore cannot and do not provide insight into how the business users and applications interact with analytic data. The legacy database monitoring tools, due to their heritage of being designed for transactional database systems and for the use by system DBA s, do not provide adequate analysis of data usage (at an object level) in any meaningful way that can be consumed by data architects, application managers, data warehouse developers and data management executives. We have too much of the same information that typical DBA tools provide. But that is not useful to understand what our business users are doing and how data is used. Director of Data Management Large Financial Services Organization. Most analytic application vendors provide audit logs to monitor application activity. They are designed specifically for application administrators to monitor activity for document management, change tracking, security administration and job scheduling. Since application audit logs are limited to activity related to the application servers, they do not correlate user and application activity with data usage and impact on the data warehouse servers, and therefore cannot provided any visibility into how data is being used by the business. 5

6 Visibility Needed to Increase Efficiency and Reduce Costs Data Warehouses in large enterprises are now routinely growing into the tens and hundreds of terabytes, and data management costs and complexity are growing exponentially. Organizations cannot continue down the path of supporting this growth without a prohibitive impact on resource and infrastructure costs. Just as business units rely on data to make informed decisions that drive the profitability and cost controls, IT organizations have to assess how the business is utilizing (or under- utilizing) the application and data assets to make informed decisions. However to increase operational efficiency and reduce the cost of managing data requires the collaboration of several functional groups within IT. Every one of the functional groups, need to be in lock step with the evolution of the BI and Data Warehouse deployment to ensure optimal performance and to control support costs. Unfortunately, legacy database monitoring tools are primarily designed for system database administrators and do not provide the necessary instrumentation for senior data managers, data architects, application managers and data integration professionals. Senior Data Managers With exploding data volumes and increasing demands from their business users, senior managers responsible for data delivery and management have the unenviable task of maintaining service levels while controlling costs. Too often the lack of insight into how the business is actually consuming and levering the assets leads to inefficient use of human resources and costs and wasteful expenditure on costly hardware and software. Senior Data Managers need the ability to: Find who/when/how strategic data is used. Measure utilization and consumption? Assess how activity is changing. Identify underutilized or unused data and application assets. Trust that sensitive data is used appropriately. Benefits: Control costs of data delivery. Justify current and future investments. Increase operational efficiency. Data Architects and Data Warehouse Managers Since legacy database monitoring tools are designed for tactical operations for database system administration, data architects and data warehouse managers have very limited visibility into what data is used, what data is unused and how data usage impacts the data warehouse. Legacy database monitoring tools do not integrate with the applications to provide a user and application centric view of data usage and its 6

7 performance impact, to enable the IT team to become more operationally efficient and to focus on retaining and optimizing the most relevant data. Data Architects and Data Warehouse Managers need the ability to: Identify which Views/Tables/Columns are most used, how used and the impact on the warehouse. Find which users and reports are sources of the bad / long running queries. Ensure the right index coverage based on data usage. Discover data that is dormant or unused; Find data that is unnecessarily loaded; Identify business critical data not loaded in a timely manner. Benefits: Optimize the data warehouse from a user perspective. Improve data architecture based on user activity. Retain and optimize the most relevant data. Application Managers Business Intelligence (BI) applications have become pervasive within the enterprise and application managers are tasked to support thousands of users with increasing demands on application performance. Legacy tools do not integrate database metrics with the application stack and therefore cannot provide an end-to-end view that is necessary for BI Application managers to diagnose and respond to end-user complaints and performance bottlenecks in a timely and efficient manner. Application Managers need the ability to: Show how poor database query performance affects business critical reports. Find who runs reports with bad queries such as unconstrained queries. Discover reports that generate frequent functions/operations better suited in DB. Identify who is using sensitive data inappropriately. Benefits: Respond faster to end-user complaints. Improve BI application performance. Meet audit requirements for security or regulatory requirements. 7

8 Business Usage Monitoring and Best Practices Business Usage monitoring is a process of monitoring and assessing business activity and data usage to deliver increased operational efficiency, ensure scalability of both infrastructure and available IT resources and reduce data management costs. For example, recently a large healthcare organization was able to analyze data usage to find out that 87% of their data was not being utilized (when reviewed over a 3 month period) and just 2 schemas out 5700 received more than 60% of their query workload. This allowed them to start developing a process to focus optimization on what was most relevant (and frequently used) and begin to streamline data loads by eliminating what was not needed. They suspended an infrastructure upgrade saving several million dollars in wasted spending - and instead began focusing on getting more out if their existing infrastructure. The goals of business usage monitoring should be to: 1. Justify Costs, Prioritize and Invest resources Based on Business Utilization With Business intelligence and data warehousing becoming mission critical for organizations, IT teams continue to come under growing pressure to deliver sustained value to the business with shrinking IT budgets. Faced with growing data volumes and increasing business demands for faster and more relevant information, enterprise IT should clearly measure and evaluate how the business is utilizing existing investments to help justify costs, prioritize resources and make informed investments. By tracking and measuring business activity, utilization and data usage trends, organizations can assess and identify underutilized IT assets along with highly utilized but underperforming assets to reduce operational costs and efficiently plan for future capacity and resources. 2. Retain and optimize the most relevant data and processes As data volumes continue to explode and business users clamor for access to more data, IT teams need to understand how users interact with data to ensure the most relevant business information is optimized and made readily available. Organizations should track and assess what data is being used and what data is not being used but unnecessarily loaded and maintained. Additionally, IT organizations should understand how data is being used by the business to focus optimization efforts on data that is most relevant for the business. With insight into how data is being used, IT services can be better aligned with the business while reducing the costs of storage and data management. 3. Respond faster and ensure scalability and performance Business Intelligence (BI) and data warehouse systems are growing in data size and complexity, and must be continuously available to support diverse, often global user communities. Business users expect that IT teams proactively discover and respond to issues before critical business services 8

9 are adversely affected. Instead of managing the BI, data warehouse and data integration stacks as independent silos, organizations should provide end-to-end visibility to the multi-functional teams responsible for data delivery. By deploying a solution that provides an integrated view into the activity of BI users and applications with a correlated view into data usage and performance impact on the data warehouse, IT organizations can gain significant improvements in operational efficiency by reducing the time and effort required to diagnose and resolve end-user complaints and performance bottlenecks. Best Practices Simply tracking usage metrics is futile if one is looking for tangible benefits in managing the BI and Data warehouse environments. Implementing a process to analyze and take meaningful action based on the analysis can deliver immediate and quantifiable benefits. Here are a few best practices to deliver quantifiable results with Business Usage monitoring: Develop key performance indicators (KPIs) to expose business utilization and consumption Too often, IT organizations rely on intuition and gut instinct to make key decisions relating to hardware and software investments as well as performance optimization. By measuring and analyzing usage activity from a business-centric view, IT organizations not only can develop key performance objectives and metrics, but can measure the results and variances, as well. Delivering actionable information across the IT organization lets the support staff work more efficiently to achieve strategic and tactical objectives. It also lets senior IT managers measure the effectiveness of their investment, drive initiatives that can gain the most business value, and reduce the total cost of ownership. Identify unused and unnecessary data to streamline data loads and archive data Since business users are constantly demanding more data, it is imperative for IT managers to assess and identify data that is being unnecessarily loaded everyday (and often many times a day) into the data warehouse but is not used or required. By identifying dormant data (i.e. Schemas, Tables and Columns), IT can collaborate with the business to develop a more efficient process of sourcing only the necessary data thereby streamlining data loads and indirectly improving data load times as well. In addition, by identifying dormant data, IT organizations can develop a plan for archiving historical data into lower cost infrastructure. For example, a large pharmaceutical company has been able to reduce data management costs by over $500,000 annually by only retaining data that is relevant and used by the business and retiring unused data. In addition they continually streamline their data loads by pruning unnecessary data and have reduced their batch load times by 50%. 9

10 Optimize and tune databases based on data usage A significant challenge database administrators face is lack of knowledge about how columns of data are actually used. Very often, indexes are created to tune the database, based on evaluating explain plans of individual SQL statements. This approach is misleading because data warehouse query workload is generally ad-hoc in nature. By identifying frequently used data that would benefit by indexing (for example, columns most often used in Where, OrderBy or GroupBy ) the database administrators can focus on indexing strategies that provide the most efficient performance. For example, a leading financial services organization saved weeks of time and effort by implementing indexing strategies based on how columns of data are being used rather than reacting to individual database SQL statements. They leverage usage monitoring and associated analytics to periodically identify frequently used columns of data that are used in a manner that would require indexing and have this information to proactively notify the relevant data warehouse administrators to take appropriate action. Reduce complexity to improve scalability and performance By analyzing how users and analytic applications interact with data, the IT teams responsible for data delivery can implement strategies to reduce complexity and improve end-user experience. Applications performance can be significantly improved by identifying frequent data conversions or functions performed by the applications that are better suited to be performed on the data warehouse. Frequent and expensive data aggregations can be discovered that are better suited to be performed during the data loading stage. Complex queries that have inordinate number of Joins, Sub Queries etc. should be identified to drive the re-design and modifications for analytical reports to reduce the impact on data warehouses. Identify poorly written reports, for example, reports generating unconstrained queries, must be identified and remedied. For example, a large manufacturing organization that supports over 20,000 analytical reports used by over 10,000 global users, analyzes how analytical applications and reports interact with data during the test and development stages to optimize end-user experience and performance. They also monitor usage on production systems to compare data usage and associated performance characteristics against benchmarks developed in test and development to rapidly discover areas of optimization. The manufacturing organization estimates savings of over $2 million in costs by improving aggregated processing times by over 22,000 hours which directly impacts business user experience and productivity as well as ensuring scalability of existing investments in infrastructure. 10

11 Appfluent Visibility for BI and Data Warehousing Appfluent Visibility for BI and Data Warehousing provides insight into business activity and data usage to manage the data warehouse and data integration processes more efficiently, help justify costs, and prioritize resources based on data utilization. The software provides detailed insight into data usage, data dormancy and workload metrics by nondisruptively capturing and correlating business user activity, application data, database information and data integration processes. The product uniquely integrates with the BI Applications such as Microstrategy, SAP Business Objects, IBM Cognos, Oracle BI and as well as providing BI application managers, Application DBA s, Data Architects, and ETL Managers, visibility into BI, Data Warehousing and ETL via a single solution. Appfluent Visibility and Teradata Viewpoint Appfluent Visibility for BI and Data Warehousing complements Teradata Viewpoint by providing insight for Data architects, Application DBA s, BI Application Managers, Data Delivery Managers, Data Warehouse developers and Data management execs. The solution is designed to provide object-level historical analytics into business activity, data usage and performance to manage data more efficiently. Teradata Viewpoint is mainly designed for Teradata System DBA s and provides visibility into database systems such as CPU, Memory, IO and individual SQL Statements etc. Key Features of Appfluent Visibility that complement Teradata Viewpoint: 1. Business Activity Analysis Appfluent provides ability to map user activity to organizational and business hierarchy to measure trends and patterns of activity, data usage and performance impact with a business centric view i.e. by business units and departments. 2. Data Usage Analysis Appfluent provides object level analysis to identify what data is used and unused and how specific Tables and Columns are used to enable retaining and optimizing the most relevant data. 3. Variance Analysis Appfluent provides the ability to compare activity between two time periods to discover variances in data usage and performance impact that impacts business services. For example, find Users (and applications and data) used in the current month that consumes 25% more database resources compared to the last 3 months. Appfluent Visibility also provides ability to compare two time periods to identify new and inactive objects. 4. BI Application and User Activity Analysis 11

12 Appfluent integrates with leading BI applications to correlate data usage and performance metrics on the data warehouse with BI User_Ids, Report names and BI data model. For example, if a BI application connects to the Teradata system as a single user, Appfluent can easily associate data usage and SQL metrics with individual application User ID s and reports on the application tier (without requiring Query Banding). 5. Teradata BTEQ Analysis Appfluent provides the ability to analyze Teradata BTEQ queries at an object level that are candidates for optimization. For example, find BTEQ queries that includes references to Table A and Table B, that has more than 5 Sub Queries and takes 10 mins in elapsed time. For more information on Appfluent visit Conclusion Data Warehouses in large enterprises are routinely growing into the tens and hundreds of terabytes and the associated data management costs and complexity are growing exponentially. Organizations cannot effectively support this growth without prohibitive impact on costs associated with resources and infrastructure. With adequate Business Usage Monitoring enterprises can gain visibility into business activity and data usage to deliver increased operational efficiency, ensure scalability of both infrastructure and available IT resources and reduce data management costs... To gain tangible benefits a comprehensive solution for usage monitoring should be combined with best practices to analyze and take meaningful action. In addition, effective usage monitoring requires a solution that integrates with the BI, data warehouse and data integration stacks to provide a complete view into business activity and data usage. Appfluent Visibility for BI and Data Warehousing is a software solution that monitors how business units and departments use data so that IT organizations can improve operational efficiency and improve scalability and performance and control data delivery costs. 12

13 Corporate Headquarters: 6001 Montrose Road, Rockville Pike Suite 1001, Rockville, MD USA Phone: (301) Copyright 2011 Appfluent Technology. All Rights Reserved. Other product and service names are property of their respective owners. APPFLUENT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. Appfluent Technology 6001Montrose Road, Rockville, MD