Optimizing the Performance of Your Longview Application François Lalonde, Director Application Support May 15, 2013
Disclaimer This presentation is provided to you solely for information purposes, is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Many factors can materially affect Longview's product development plans and the nature and timing of future product releases. The development, release, and timing of any features or functionality described remains at the sole discretion of Longview. This information may not be incorporated into any contractual agreement with Longview or affiliates. Despite the continued efforts of Longview, its suppliers, and/or sponsors to ensure that the information in this document is as complete and up-to-date as possible, Longview, its suppliers, and/or sponsors cannot warrant the correctness and/or completeness and/or specific applicability of the published and/or requested information in this document. Longview, its suppliers, and/or sponsors shall not be liable for any direct, indirect, incidental, special or consequential damages, lost profits or for business interruption arising out of the use of this document. The extraction and use of information from this document remains at all times completely within the user s own risk. No part of this document may be reproduced, published and/or transmitted in any form or in any way, electronically, in print, in photocopy, on microfilm or by any other means, without the prior written consent of Longview, its suppliers, and/or sponsors. Exact International Development B.V., 2013. All rights reserved. All trademarks mentioned herein belong to their respective owners. 2
Agenda Session Overview Khalix / Longview 7 Architecture Review Design Considerations When Upgrading to Longview 7 Khalix / Longview 7 Performance Areas Longview 7 Performance Improvements Longview 7 Optimization Methodology Optimization Tools and Features in Longview 7 Other Optimization Considerations Q&A 3
Architecture Khalix (V3) 4
Architecture Longview (up to 7.1) 5
Architecture Longview 7.1 Update 3 Dashboard Application Server Smart Client Web Server Database Repository Clients Web Services Data Servers Data Server Repository Application Framework Web Services SharePoint Web Part & BCS/ECT 6
Khalix / Longview 7 Architecture Understanding the Khalix / Longview 7 architecture What makes it so unique? Khalix / Longview 7 is a solution comprised of an intelligent dataset that is highly configurable Data structure is truly dimensional, n_d1 x n_d2 x n_d3 x n_d[n] Symbols, hierarchies, structures are defined by customer Data server has accounting intelligence» Rollups such as +, -, Carry Forward» Foreign Exchange, Intercompany, Journal Entries Khalix / Longview 7 clients and interfaces make the other half of the solution Complex business functionality (models, procedures, services) Application Framework (Modeling node before Longview 7.1.1) Configuration is tied directly to the business processes and requirements 7
Khalix / Longview 7 Architecture Understanding the Khalix / Longview 7 architecture What makes it so unique also makes it challenging Solution is tailored to business processes and requirements Very unique for every customer Data structures are created and maintained by Business Team IT impact is not always considered when making design choices Dimensional nature of the dataset impacted by structural changes Business process and technical processes can conflict after major structural changes e.g. Restatement times 8
Khalix / Longview 7 Architecture Extended scalability in Longview 7 Item Description Khalix Longview 7 # of Dimensions 8 16 # of symbols per dimension 32,000 256,000 Symbol name length 10 char 31 char Symbol / dimension / attribute naming convention Leading alpha, no specials No leading alpha,. (period) and _ (underscore) are valid String length e.g. comments 100 characters 4,000 characters 9
Khalix / Longview 7 Architecture Khalix / Longview 7 Dataset Size & Structures Not one factor is directly related, but rather a combination Number of symbols is not the only telling factor Number of parent symbols is a factor Number of parent-child relationships (e.g. alternate hierarchies, depth of hierarchies) is a factor Data density definitely is a factor (e.g. allocation processes) Server rules and accounting logic used Practically impossible to predict size 10
Design Considerations Longview 7 considerations Conversion from Khalix to Longview 7 Made easier with Khalix clients brought forward Longview 7 offers unprecedented analysis opportunities Increased number of dimensions (up to 16) Increased number of symbols per dimension (up to 256,000) Increased limit for NDD patterns Hierarchy synchronization and many more features Application Server demand may be larger Tied to configuration based on business requirements and Longview 7 functionality used 11
Design Considerations Longview 7 considerations What are the performance implications of using the extended symbol and dimension limits? Design stage is the best time to take Longview 7 architecture and tuning features into consideration Adding more Accounts or Time periods would relatively expand dataset Adding parents in new, outer dimension could exponentially expand dataset Without advanced optimization tools and features the dataset could be almost impossible to manage Recent test with seven dimension application: With Virtual Symbols: 9.2 billion UPN records» Restatement Time: 4 h 25 m Without Virtual Symbols: 1.9 trillion UPN records» Restatement Time: 93 h 30 m*» * there was no write phase (special binaries) 2,000 1,000 0 UPN Records (in billions) No Virtual Symbols Virtual Symbols 12
Design Considerations Longview 7 considerations It may be necessary to reassess the hardware if: The number of dimensions, symbols and hierarchies change Logic used changes (e.g. server rules, accounting logic, etc.) The number of users changes How Application Framework logic will be used in Longview 7 13
Khalix / Longview 7 Performance Areas Application Performance Performance related to four main areas Query times are typically longer with larger datasets Submission times based on number of resulting derived records Restatement time tied to number of derived records, functionality used and server rules Longview Application Framework processing Dataset size directly impacts performance of application Size of tables affect RDBMS performance Size directly impacts server load (CPU, RAM) 14
Khalix / Longview 7 Performance Areas Restatement / Partition Recalculation Batch process completely recalculates all derived data Hierarchies, rollups, server rules used to reproduce the derived data based on source data records Main restatement consists of two phases: Math Phase Math servers split up work based on hash value to calculate derived data Data stored into temporary files Math phase is very CPU intensive, benefits from speed and quantity of processors Write Phase Gathers and sorts temporary files into load files for loading into the RDBMS or into data cache files Disk I/O intensive, data throughput is a major factor Additional phases if elimination and / or journal entry logic is used A partition recalculation is a recalculation of a subset of the database as defined by the Khalix / Longview 7 partitioning setup 15
Khalix / Longview 7 Performance Areas Application Framework for server side modeling Features in Khalix Calculates data in n-dimensions Triggers calculations based on submission events (i.e. event triggers) Trigger calculations in input templates Trigger processes via web links Work performed on the Application Server Utilize one CPU on the Application Server per AF process CPU utilization can be up to 100% Additional features in Longview 7 Calculates data in n-dimensions n-dimensionally Import or export data via bridges with mapping tables Hierarchy synchronization 16
Khalix / Longview 7 Performance Areas Comparing the performance of Khalix to Longview 7 All things being equal : Upgrade with the same number of dimensions, symbols, logic used, server rules used, etc. Restatement performance is better in Longview 7 Query performance is better in Longview 7 Submission performance is comparable Application Framework performance is improved in Longview 7 17
Longview 7 Performance Improvements 18
Longview 7 Optimization Methodology The keys to successful performance optimization are: Determining what the performance issues are and where they are occurring Defining clear and specific performance objectives Tracking of measurable performance elements How performance tuning is done is as important as what is done Use of a methodical, repeatable and comprehensive tuning approach Understanding the factors that impact performance Understanding the measures that can be used to modify performance 19
Longview 7 Optimization Methodology Define clear and specific performance objectives Select benchmarks Benchmarks must be representative of how the users utilize the Longview application Benchmarks must be comprehensive, covering all aspects of the Longview application Benchmarks must be simple, substantial and unique Client-side versus Server-side processing Include well-performing processes as well Select timing measurements Longview monitor file (lv_dataserver.log) information Query statistics (in lv_dataserver.log file) Longview Desktop commands: History On / Off, Timerrun, Timerstart / Timerstop Database Audittrail command Batch information Stopwatch 20
Optimization Tools and Features Areas of Tuning & Optimization Application design Longview 7 performance features RDBMS Hardware Operating System Hardware Processors Memory Disk subsystem Network Virtual Environments Application Design (dimensions, hierarchies, rollup rules, virtual symbols, application/custom code, etc) Tuning Features (partitioning, miscellaneous server configuration parameters, etc) RDBMS (indexes, tablespace layout, etc) Operating System (CPUs, memory, I/O, etc) Diminishing Returns 21
Optimization Tools and Features Several general measures can improve the overall performance of Longview 7 Remove unnecessary hierarchy rollups Use the Longview Export / Import functionality to optimize the database Limit the use of alternate hierarchies Limit the scope of PAC / YTD functionality Optimize the use of server rules Use event triggers and / or query rules instead of server rules when possible 22
Optimization Tools and Features Managing the dataset size Longview 7 design (i.e. the business design) is where good performance starts Consider the design, consider the business process Configuration features can dramatically affect performance Four main optimization tools: Virtual parent symbols Longview 7 partitioning Longview 7 data cache functionality Optimized configuration parameter settings (in lvsrvr.cfg file) Additional optimization tools: Grid restatement Non-standard data table index order Rollup Rules 23
Optimization Tools and Features Virtual Parent Symbols Overview Parent data is only calculated dynamically when required for a query, not during a restatement, nor during dynamic submissions Reduced restatement time and data submission time If configured correctly, maintain or improve query performance Allow for greater partitioning flexibility Important feature for controlling potential data growth with use of more dimensions and symbols in Longview 7 Considerations Too many virtual parents can cause poor query times Virtual symbols should be set by applying changes incrementally, using a methodical approach to gauge the impact of changes 24
Optimization Tools and Features Virtual Parent Symbols Strategies Set to virtual parents in entire dimensions where parents and leaf symbols are most often queried together (e.g. ACCOUNTS and TIMEPER) Set to virtual parent symbols in alternate rollups Set to virtual parent symbols used infrequently in reports / queries or used by only a few users Use a layered virtual parent strategy (e.g. set virtual parents in alternating hierarchy levels) Set to virtual parent symbols with three or less children, in the bottom levels of hierarchies With best available hardware and with optimized hardware setup, an aggressive virtual parent strategy can be used 25
Optimization Tools and Features Longview Partitioning Overview Distribution of data across multiple tables and Longview data cache files in a way that is meaningful to the application level (i.e. logical partitioning, not to be confused with RDBMS partitioning) Typically results in better read and write concurrency Provides the ability to run recalculations for specific partitions, allowing users to continue using partitions not impacted by the recalculation With the increased number of dimensions in Longview 7, partitioning at a more granular level can be an effective way of managing the potential data growth 26
Optimization Tools and Features Longview Partitioning Strategies Separate hierarchies based on the type of data they contain (e.g. time periods, versions, etc.) Separate hierarchies so that data is evenly distributed between multiple tables and Longview data cache files, to improve read and write concurrency Separate hierarchies based on user activity, to allow for partition recalculations and better application concurrency Example: Partitioning the TIMEPER dimension across category years (e.g. Actuals 2009, 2010, 2011, etc.) 27
Optimization Tools and Features Longview Partitioning Example Using the time period split example The TIMEPER dimension could be partitioned by calendar year (e.g. 2010, 2011, etc.) The CATEGORIES dimension could be partitioned by category (e.g. Actuals, Budget,etc.) The MEASURES dimension could be partitioned by measures (e.g. YTD, MTD, etc.) 28
Optimization Tools and Features Longview Data Cache Functionality Overview All data calculated by a restatement or partition recalculation is stored in binary files on the Application Server instead of in database tables Primary benefit is optimized disk space usage Usually improves restatement and query times Source / leaf data and metadata are always stored in the database tables Data calculated dynamically during the period between restatements or partition recalculations is stored in the database tables Queries to retrieve calculated data are performed against both the data cache files and the database tables and the results are merged before returning the rows to the client application 29
Optimization Tools and Features Longview Data Cache Functionality Considerations Too much data stored in the database tables when Longview data cache files are used may result in slower queries if the Longview 7 servers have to spend a significant amount of time merging data from the database tables with the data stored in the data cache files Restatements or partition recalculations should be performed as frequently as required to ensure the Longview data cache files are up-to-date 30
Optimization Tools and Features Restatement optimization Optimization options Balance the Math Phase Optimize the MATH_POOL, MATH_SERVER_RAM, RECALC_HASH_DIMENSION and RECALC_HASH_VALUE settings in the lvsrvr.cfg configuration file Virtual parent strategy Longview partitioning Hardware Processors (quantity and speed) Memory Disk subsystem Grid restatement options Longview 7 grid implementation DataSynapse grid implementation 31
Optimization Tools and Features Longview Server Parameters Configuration COMPRESS_TEMP_FILES When the temporary file compression logic is enabled, the temporary files generated by the restatement are compressed to minimize disk storage requirements, resulting in a slight performance impact on the Math and Write phases of the restatement Set to FALSE for best performance RECALC_HASH_DIMENSION Sets the dimension used to calculate data during the restatement With the default value of AUTO, the restatement uses the dimension with the largest number of symbols In the majority of applications, AUTO is the best setting If the restatement Math phase is unbalanced, explicitly set to a dimension with better data distribution Can help reduce file overlap during the Write phase 32
Optimization Tools and Features Longview Server Parameters Configuration RECALC_HASH_VALUE Determines how the leaf data is divided (hashed) for distribution to the Longview math servers during a restatement An optimal setting helps balance the Math phase of the restatement If leaf partitioning is not used, setting to a value between 72 and 96 typically provides best results If leaf partitioning is used, setting to a value between 16 and 24 typically provides best results MATH_SERVER_RAM Sets the maximum amount of physical memory each math server can use, in kilobytes Can help balance the Math phase of the restatement and helps reduce file overlap during the Write phase Setting to a value of 262144 (256 MB) typically provides best results 33
Optimization Tools and Features Longview Server Parameters Configuration WRITER_ASYNC_IO and WRITER_SERVER_RAM WRITER_ASYNC_IO specifies whether to use asynchronous writes during the Write phase Set WRITER_ASYNC_IO to TRUE for best performance WRITER_SERVER_RAM specifies the amount of memory (in kilobytes) reserved in the KSA for the writers when the WRITER_ASYNC_IO functionality is used Set WRITER_SERVER_RAM to a value between 204800 (200 MB) and 262144 (256 MB) for best performance Only available for Khalix / Longview running in a Windows environment MATH_POOL Sets the number of math servers available to process jobs in the math queue For best performance, set MATH_POOL to a value equal to the number of CPUs / cores available on the Application server 34
Optimization Tools and Features Longview Server Parameters Configuration WRITER_POOL Sets the number of writers available to process jobs in the writer queue If Longview partitioning is not used, set to 4 If Longview partitioning is used, set to a value between 6 and 10 for best performance AGENT_POOL Determines the number of agents available to handle connection requests from the various non-web Longview clients A value of 0 disables the agent pooling logic; this is not recommended For best results, set to a value representing one agent for every four expected concurrent users; for example, if 100 concurrent users are expected, set to 25 35
Optimization Tools and Features Longview Server Parameters Configuration WEB_AGENT_POOL Determines the number of Web agents available to handle user connection requests via the Internet/Intranet A value of 0 disables the Longview Web functionality For best results, set to a value representing one agent for every 10 expected concurrent users; for example, if 100 concurrent users are expected, set to 10 SUBMISSION_POOL Sets the number of submission servers available to process source / leaf data submissions If leaf partitioning is not used, set to 2 If leaf partitioning is used, set to a value between 2 and 6 for best results GATHER_ORACLE_STATISTICS If the RDBMS is Oracle, set to TRUE With this parameter set to TRUE, Oracle statistics are gathered on the temporary tables created by the Longview servers at server startup 36
Other Optimization Considerations RDBMS considerations Compared to Longview 7 optimization features, RDBMS tuning typically provides diminishing performance returns Ensure optimal setup of database (e.g. tables, indexes, etc.) Ensure database statistics are current Disk I/O optimization is key Use RDBMS performance monitoring tools to monitor Disaster / Recovery strategy may impact performance 37
Other Optimization Considerations Hardware considerations Processors Quantity and speed directly impact restatement times Also impact submission math and virtual parent processing Intel and AMD platforms currently have the advantage with regards to restatement times Disk arrangement Type of disks (e.g. internal, SAN, SAS, etc.) / RAID levels can affect Longview 7 performance: Can have significant impact on dynamic submission times Can have significant impact on restatement times Data Server setup one tier versus two tier Dependent on each situation, such as concurrency, server location, connection between servers One is not necessarily better than the other 38
Other Optimization Considerations Virtual Environments Longview 7 considerations Longview 7 is highly configurable and potentially complex, depending on how it is implemented; the more complex it is, the more it benefits from having the fastest resources available to it With expanded dimension and symbol limits, Longview 7 datasets may require an aggressive virtual parent strategy, therefore the Data Server will require optimal processing and disk performance Internal testing and customer observations have shown that restatement and other Longview 7 performance can be materially affected when running within a virtual environment 39
Other Optimization Considerations Virtual Environments Virtual environment considerations In our experience with internal testing and with observations at our customers, disk I/O has been observed to be as much or more of a bottleneck than virtual processors in a virtual environment Most customers have VM software that limits to 4 virtual processors During day-to-day monitoring Longview 7 can sometimes appear to be using few resources however in peak times having full access and use of machine resources is critical Over-allocation of processors at the host level is not recommended for virtual machines running Longview 7 Performance issues are more difficult to trace because the root cause can be invisible to application monitoring Monitoring resources utilization at the host level is important 40
Thank You! Questions?