Reference Certificate Vendor name: PRISE Recipient: Telenor Hungary Reference issuer: Zsolt Kovács, Head of Infrastructure Operation Subject of service: PRISE Compress Wizard implementation Vendor introduction PRISE enables its customers to realize the maximum value of their Teradata investments by decreasing storage space and hardware consumption using a proprietary self developed methodology. After a free analysis of customers hardware usage patterns, PRISE can rapidly optimize storage space and hardware consumption. Achieved benefits significantly reduce cost of ownership. Recipient Telenor Hungary 2nd largest Mobile Network Operator 3.6 million active subscribers Net 9.4 TeraBytes Teradata Data Warehouse Content of service PRISE successfully implemented PRISE Compress Wizard at Telenor Hungary in Q2 2011. Implementation milestones: Analysis of current data storage Report the forecasted storage and performance impact Implementation of storage optimization Final report of the real storage and performance impact PRISE business model is simple and effective: analysis and forecast reports are free. Savings are shared upon measuring the real impact. Achievements Analysis performed without any congestion in regular processes Reports forecasted a possible 1200 Gbytes of net space gain Implementation performed without any operational issues and released 1258 Gbytes of net free space, which is 17.0 % of the affected tables net storage space 15.5% of the affected databases net storage space Suitable for 11 months data increase Optimization had no measurable performance impact on load procedures Optimization caused a total of 13% decrease of overall IO and 15% of overall CPU usage The project delivered cost reduction in the value of ~100.000 EUR within 1-2 weeks* (*extended by phasing and resource availability) Impressions The vendor demonstrated excellent knowledge in Teradata and related technology PRISE Compress Wizard implementation delivered real value to Telenor Based on experience I highly recommend PRISE and it s products as a professional information services partner to your company. Budapest, 2012.03.02. Zsolt Kovács, Head of Infrastructure Operation ncsay Dániel, Director of SSOD
Case Study The Companies PRISE - Vendor PRISE enables its customers to realize the maximum value of their Teradata investments by decreasing storage space and hardware consumption using a proprietary self developed methodology. The implemented PRISE Compress Wizard delivers quick win results for customers by immediately releasing resources like storage space and processing capabilities, and decreasing the specific operational costs. PRISE s experience shows that even the best managed Teradata environments hide significant optimization opportunities. Unique enablers of the company are the over 10 years of Teradata Masters industry experience and the proven solution for optimizing Teradata database internal compression. PRISE Compress Wizard implementation projects demand minimal onsite activities, have a short implementation time, therefore offering quick wins. Most of the benefits manifest as immediately usable free database space and increased processing capacity, while others reduce the long-term costs. For more details please refer to Annex A. Telenor Hungary - Recipient Hungary s 2nd rank mobile network operator has been founded in 1994, and fully owned by the Telenor Group since 2002. The company s telecommunication services cover voice, text message (SMS) and mobile internet over 2G and 3G network. The 3.6 million subscribers distribution is: 52% postpaid and 48% prepaid. The company achieved 650 million EUR revenue in 2009. http://www.telenor.hu/en/telenor-hungary/facts-figures/ http://www.telenor.com/ In 1999 the firm decided to build a Teradata based Enterprise Data Warehouse (EDW) solution. The solution has since been continuously developed according to best practices to fulfill an ever expanding set of enterprise requirements. It has been deemed the best data warehouse solution in the Telenor Group in 2006, due to the complex analytical functionalities it provides, such as profitability calculation, social network modeling and churn prediction, and the one to one marketing capabilities established on top of it with the implementation of a campaign management solution. The data warehouse of Telenor Hungary stores a nearly 360 view of the customers and the company and stores all detail data for 13 months and aggregated level data from inception. During its lifetime the EDW has grown to 5 times its original size, and at least 3 times in complexity and the EDW has also been integrated into most core business processes, making it one of the most important IT systems within the company.
Methodology Project lifecycle Analysis Implementation Technical Methodology and preparation Forecast report Result report and closure Business 1-2 weeks 1-2 weeks Decision Figure 1: PRISE Compress Wizard implementation lifecycle Methodology and preparation Non-disclosure agreement signed. Technology demonstrated on-site including ways of working, project milestones and timelines, contractual highlights and guarantees, Q&A. Technical environment initialized, including access permissions, resource allocation, timing. Roles: Vendor: On-site visit, presentation, requirements definition Recipient: DBA and decision-maker get familiar with the product, resource planning, technical preparations Analysis Target areas identified. PRISE Compress Wizard assembled according to requirements and run. PRISE evaluates results within 3 working days. Roles: Vendor: Consults, prepares and supervises product configuration and scheduled run Recipient: Participate in identifying focus areas, available for assistance
Forecast report Findings and recommendations prepared and presented. Q&A session. Roles: Vendor: Calculates forecasted benefits, creates reports and perform presentation Recipient: DBA and decision-maker get precise information about the achievable benefits Decision Recipient makes decision about implementing PRISE Compress Wizard optimization. No-go decision concludes the project with zero costs to Recipient and implementation will not take place. Implementation Optimization scripts are provided, deployed, scheduled and run. Roles: Vendor: Recipient: Deploys optimization scripts and assists in running DBA fully involved in implementation, managing EDW processes and running optimization scripts Result report and closure Log data evaluated and the results of the overall project are assessed and presented. Project closure with benefits shared as previously agreed. Upon mutual satisfaction Recipient issues a reference certificate. Roles: Vendor: Recipient: Assesses and presents results Evaluates results
Optimization process PRISE Compress Wizard storage optimization is based on fine tuning Teradata Multi-Value Compression (MVC) functionality. MVC is a 100% transparent feature, using it does not influence the applications. MVC is usually manually configured and will not automatically adapt to the changing business and data environment. From technical point of view the project has two separate tasks: analysis and implementation. During the analysis phase the vendor s expert identifies the set of tables in focus in cooperation with the Recipient s delegated DBA. The qualified tables will be analyzed by a proprietary software - PRISE Compress Wizard, which inspects the distribution of data values, calculates the optimal MVC settings and calculates the possible space gain for each involved table. The Vendor then evaluates the result statistics and presents the suggested optimization prospects. The second technical phase of the project covers the implementation, which enables the Recipient to apply the benefits in practice. The Vendor s and Recipient s DBAs create a deploy time plan to effectively execute the suggested modifications in the database. According to the plan the Vendor delivers SQL scripts that relocate data into the optimized table structure. The scripts are specifically designed to avoid all incompatibilities or system disturbances. For details please refer to Annex B. Benefits of compression The cost structure of a data warehouse consists of several different components: Storage and processing hardware costs, e.g. servers and disk arrays License and support fees in proportion to the system performance Operational expenses like backup, space and performance management, optimization Electric power consumption of the hardware components Server room reservation and cooling costs Personnel costs Implementing PRISE Compress Wizard will decrease all cost components, saving significant expenses for the company, without noticeable disadvantages due to Teradata s unique technological solution utilized by PRISE Compress Wizard. Compression is recommended to be performed from time to time in order to exploit the maximum advantages of the product. Benefits achieved can be classified into two categories. For further details please refer to Annex A. Instant advantages Lower storage consumption, instantly freed disk space Less resource consumption -> faster queries to the users Shorter backup windows, less archive media Deferred hardware extension No side effects or application disturbances Occasional advantages Shorter migration, reconfiguration times at upgrades Shorter disaster recovery time Slower growth of database size, less frequent capacity expansions More applications and reports can be served by the current infrastructure Significantly less DBA time is required for manual database optimization tasks
Trend of: CPU time, IO, Workload Achievements at Telenor Hungary Storage space saving On the optimized tables PRISE Compress Wizard managed to reach 17% storage space saving, which resulted in 4%..92% savings on individual tables. To visualize the achievement, the gained storage area is enough to cover 11 months of data volume increase. Detailed estimations and statistics are available in Annex C. Reduction of resource consumption Both the CPU and IO load of the system has been decreased by 15% and 13% respectively, at significantly increased user activity. The chart shows the system measurement results: 0 Figure 2: Overall Workload and CPU&IO usage and trends, including optimization phases Telenor Hungary s Teradata RDBMS has the DBQL (Database Query Logging) feature switched on continuously, and stores ~1-2 months of historical data. The chart of Figure 2 is based on DBQL data and shows the total CPU and IO load of the system over the time. Phases : shows the optimization implementations in time (phase 4,5,7,8) System CPU time: 7 days moving average of daily SUM() of CPU seconds System IOs: 7 days moving average of daily SUM() of IO Workload: 7 days moving average of number of SQL statements issued daily
Project details The optimization project was initiated by PRISE in Q1 2011. It required noticeably low human resources from Telenor side, only one DBA, who prepared the working environment and then executed the implementation package. Telenor DBA time spent on the project did not exceed 5 working days. Decision maker time involvement was limited to a few hours in meetings. The time frame of the project was quite long compared to the required human and machine resources. The reason was low resource availability, which lead to 8 implementation phases. The project did not have a drawback due to the lengthier timing, all benefits were delivered. Figure 3: PRISE Compress Wizard implementation project timing
Annex A. Aspects of benefits Some of the achieved advantages will not be visible immediately, but on the long run they will undoubtedly apply: the company s growing business produces more and more data and rising market competition demands more complex algorithms, both of which require developments of hardware infrastructure sooner or later or periodically. Optimizing available resources will decrease investment into additional hardware resources along with their costs. Teradata license and support fees calculation model is based on the idea of TPERF, which is a weighted mixture of processing capacities. The more CPU and IO capacity a system has, the bigger the TPERF value of an installation, and raises the fees higher. More server nodes and disk bundles consume more electric energy. Researches demonstrated that the power consumption expenses of IT infrastructure fall into the price range of license fees. The power energy consumption additionally doubles for server room cooling. Other cost components are reduced immediately. Backup and archive times shorten proportional to freed disk capacity. Companies often suffer from the lack of free space. In this case difficult decisions must be made: investing into additional hardware or reducing either the scope of stored information or the depth of history. These will be painful to some division of the company unlike compression, where there are no tradeoffs. It is important to highlight, that PRISE Compress Wizard is 100% transparent for all kind of applications, therefore introduction or optimization of the product will not involve ETL, application or report development, which means there is no hidden cost.
B. Technical description of project Prerequisites For quality assurance purposes we prefer to use Teradata s DBQL feature. We use 3-4 weeks log data as basis of resource consumption calculation for analysis & implementation as well as final results. Target selection We used the Teradata data dictionary to make a pre-selection of the candidate tables. Based on a consultation with development, reporting and operation staff, we excluded the unnecessary databases and tables from the scope. Project decided to take the TOP 1000 tables by size to be examined, and after excluding the unnecessary and stale ones, 889 of them were candidate for analysis. Calculation process The analysis of the <target> is a 100% percent automatic process which has the following prerequisites: 1 database in the EDW, where PRISE Compress Wizard can write, and have ~1000MB (depending on local situation), referenced <CCWDB> 1 user in the EDW, which has read-only access to all EDW database tables, and data dictionary, and read-write access to <CCWDB>. The given user and enough resources (priority, TASM limits) to be able to run the statistical processes. In Telenor Hungary PRISE used a standard Developer Role to run the analysis process with its default priority, spool space and access rights Enough spool space for calculations, depending on the analysis sample size. A Windows workstation with o JVM 1.6 installed o Teradata client installed o Network access to EDW system (through JDBC) The process was run in multiple parts, during the nights, in order to minimize the disturbance of EDW users or daytime if the priority settings enable seamless EDW usage. The time the whole process takes depends on Available processing resources Number of tables in <target> Number of columns in <target> s tables Storage size of <target> The calculation process runs as an ordinary ad-hoc report, does not disturb loads (uses access locking) and other reporting and analysis tasks. The calculation process in Telenor Hungary was run with the following details: Used 1million record sample on each tables Ran in 3 parts, took 10h, 16h, 1h respectively We utilized a total of 94 CPU hours and 916 million IOs for the calculations The calculation results (including detailed value distribution data) occupied ~700Mbytes
The result of the calculation process contains the original and the estimated optimized table sizes, to be able to estimate the amount of workspace that will be necessary for implementing the optimization and the storage gain which can be achieved before any modifications are done on the database. Analysis & report After the completion of the calculation process we found that the following gains are available. The table contains only the statistics for the tables that are currently not optimal. Gain % range Orig. sum GBytes Gainable Gbytes No. of tables 0..5% 3856,85 124,57 222 05%..10% 1269,25 101,94 134 10%..20% 2388,77 336,13 213 20%..30% 864,62 227,6 137 30%..40% 624,15 215,14 64 40%..50% 369,86 175,57 48 50%..60% 126,37 68,86 30 60%..70% 36,01 22,82 27 70%..80% 21,67 16,2 6 80%..90% 11,35 9,35 7 90%..100% 2,02 1,84 1 Total 9570,92 1300,02 889 Target selection If we want to cut the time and resource demands of the implementation, it s reasonable to shorten the <target> table list. We ve chosen to eliminate those tables that did not promise more than 20Mbytes of space gain. This way we had 488 tables left for implementation phase. Implementation Table compression parameters are creation-time defined; non modifiable in the available versions of Teradata, therefore the modification of compression is only applicable by recreating the tables, which will cause object and data absence for a while. For this reason we had to choose low activity time windows to implement optimizations. The most appropriate time was the end of business time: 17:00 PRISE Compress Wizard generates all the necessary scripts that need to be run to achieve the possible storage gain by recompressing the tables, but they must be run by the local Teradata DBA. Before running we recommend that the DBA checks all the SQLs before they are applied. The implementation was broken up into 8 phases since it was quite difficult to find continuous time frames with available resources of both DBA and database.
C. Result details Space saving The total results of the 8 implementation phases are shown in the table below: Phase No of Tables Original Gbytes Gain Gbytes Gain% Runtime 1 3 183,4 72,5 39,53 ~2h 2 8 543,08 158,22 29,13 ~2h 3 5 848,67 169,92 20,02 ~3.5h 4 20 388,46 99,81 25,69 ~2h 5 406 2304,38 484,03 21 ~8h 6 1 120,67 34,07 28,23 ~0.5h 7 18 705,37 100,26 14,21 ~3h 8 27 2308,55 140,1 6,07 ~6h Total 488 7402,58 1258,91 17.0% ~27h The implementation phase requires the optimized space amount of the target table as an additional free space during the technical run of the scripts, for one table at a time. This space will be immediately de-allocated after the table s optimization finishes. The implementation consumed ~350 CPU hours ~1450 million IO operations Which are 80% and 25% respectively, of the regular daily system loads. Load performance impact The test measurements proved that the CPU consumption of an INSERT statement into a compressed table can be 50-200% more, than into its uncompressed equivalent. However, one can easily check that a suboptimal compression definition will demand the same, or more CPU. Load procedures typically have two important time factors: longer preparation & transformation and significantly shorter final INSERT, however the more resource intensive first part is not affected at all. After the implementation finished, we performed an investigation of the CPU and IO consumption of the INSERT statements issued against the optimized tables:
0,26x 0,5x 0,62x 0,74x 0,82x 0,9x 0,98x 1,06x 1,14x 1,22x 1,3x 1,44x 1,76x 1,9x 2,18x 3,98x 5,12x 7,3x 20 18 16 14 12 10 8 6 4 2 0 Data Load CPU change Number of loads Figure 4: How did CPU usage of affected tables INSERT statements change Performance impact is displayed on the horizontal axis, 1x means no change, 0,5x half, 2x double amount of CPU for the similar INSERT operation. The number of tables where INSERT phase fell into a given performance impact range can be found on the vertical axis. One can read that the modal impact was around 0.95, therefore INSERT statements consumed a little less CPU than before optimization. Fault tolerance during implementation Production Data Warehouses are in continuous operation and will not easily go down for maintenance. For this reason PRISE developed a safe process to foreclose the possibility of data damage or loss and minimize the possibility of unavailability. In Telenor Hungary s PRISE Compress Wizard implementation project we reached these goals 100%: No particle of data was lost or corrupted No database downtime was requested Personal data security PRISE undertakes to sign a very comprehensive non-disclosure agreement as a first step. PRISE Compress Wizard will not modify or keep any personal data. Additionally we only work with your data in your safe environment. Change management PRISE provides detailed documentation for ITIL change & configuration management process considerations. PRISE Compress Wizard can be tested initially in a test environment, then gradually implemented across the entire scope with maximum safety.