EMC Documentum Performance Tips Ten Rapid Checks for Improved EMC Documentum Performance Monitoring EMC Documentum for Continuous Improvement September 2012 By Eric Ryder Senior Consultant, Reveille Documentum Practice 14 years EMC Documentum Professional Services Ensuring EMC Documentum Service Levels
Introduction This document identifies immediate-impact tips for EMC Documentum performance improvement. These tips have been field tested and validated from years of hands-on experience in live production settings. Also included are suggestions for longer term application monitoring and maintenance. At Reveille Software, we re dedicated to ensuring the long-term management and optimization of EMC applications. Your EMC Documentum Application Environment is Constantly Changing EMC Documentum is a capable but complex application platform. During implementation, deployment, and subsequent production use, the configuration needs to be frequently modified for: Infrastructure changes Performance tuning Deploying new applications Updating applications Adding on new users and groups Installing upgrades and patches There are, in fact, a wide variety of factors that can, and do, lead to on-going modifications to the original deployment. This impacts performance, resource management, and optimal configuration. Maintaining optimal performance is an ever-moving target. A Proper Performance Audit and Engagement Takes Time and Resources Executed properly, a performance audit can take days or even weeks, depending on the deployments complexity. Further, several in-house and/or third party resources need to be involved to get a comprehensive picture of system configuration and performance; this can include database administrators, network personnel, system administrators, application developers, support personnel, and Documentum administrators. Finally, proper diagnostic tools and a deep understanding of how Documentum applications interact with each other is required to complete the discovery, and recommend solutions. Documentum Performance Tips Reveille Software Page 1
The Database team, who are managing other competing priorities, must also be on board. Database settings need to be investigated, query plans generated, and queries built for discovery. On the web tier, it takes the proper tools and training to run traces and fine tune important settings like heap size and configure for load balancing. Typically, customers do not have these required tools or the know-how to use them, and resort to trial and error. From a networking perspective, the network team must verify and adjust for proper connection speeds between Documentum components. Often, upon an in-depth investigation at the network interface or switch level, the configuration settings are not optimal for best performance. Trial and Error is neither efficient nor cost effective With a lack of time, resources, and tools, customers often resort to trial and error. They try many different configuration values, add unnecessary indexes, throw more hardware resources at the problem, and hope for an improvement. This has three major issues; one, it is not very productive; two, there is not an empirical way to measure if the trial-and-error process worked because customers lack objective performance measurement tools; and, three, this is not cost effective. Quick Hits: 10 Performance Improvement Tips for EMC Documentum There are many settings and parameters that are unique to every Documentum implementation. A thorough evaluation, including consulting sessions with the various teams, is required to properly assess and optimize a given implementation. At a high level, what follows are ten best practices to ensure that you are on your way to optimized performance. Documentum Performance Tips Reveille Software Page 2
Performance Setting Why it is Important What Is Often Found This setting can have a Default for this setting is dramatic effect on how EXACT. This is a poor choice the optimizer parses for Documentum databases. queries. 1 If Oracle, make sure cursor_sharing is set to FORCE or SIMILAR. 2 If Oracle, make sure optimizer_ind_cost_adj is set to 5. This setting can have a varying effect on how the optimizer parses queries. Though there are more definitive ways to determine this value, 5 has been recommended by Documentum as a good starting point. Default is 100. 3 If Oracle, make sure optimizer_index_cache is set to 95. 4 Make sure heap size is sized properly, preferably using JMeter, or some other tool that can show garbage collection. Typically, -Xms and -Xmx should be 1024mb. This setting can have a dramatic effect on how the optimizer parses queries. Garbage collection has a huge effect on performance. This setting has the biggest effect on the optimizer. Default is 0. Setting this either too high or too low will cause excessive thrashing or too long between releases of memory. 5 If SQL Server, when using SQL Server for larger volume deployments, (many millions of rows in dm_sysobject), the following parameter settings have been proven to greatly enhance performance. Max Degree of Parallelism = 4 PARAMETERIZATION = FORCED AUTO_UPDATE_STATISTICS_ASYNC = ON AUTO_CREATE_STATISTICS = OFF These settings can have a dramatic effect on how the database handles queries involving dm_sysobject. Default settings for these parameters are not suitable for large-volume implementations, (>10 million objects). Documentum Performance Tips Reveille Software Page 3
Performance Setting Why it is Important What Is Often Found This affects scale: if you have a network switch set to 100mb, it is the weak link in the chain and setting it to 1GbE will make everything 10 times faster, (relatively speaking). 6 Ensure gigabit speeds between all Documentum servers, including all network switches in between. Often, all network interfaces (NIC) are set properly to 1GbE, but a rogue switch somewhere is set to 100mb. It may only get saturated at peak times, making this bottleneck hard to find. Often the network teams report that 1GbE is the default, but somewhere along the line a change has been made. 7 The connection to database storage should be fiber, or similar speed, and dedicated. There is a high volume of chatter between the Content Server and database. Often, the database is just a server hanging off the network, like any other server. If there is not a very fast, preferably dedicated, pipe between the CS and DB, there will be a big impact on query performance. 8 CPU and memory usage should be monitored during "hot" hours, where there is the highest concurrent user activity. Sizing should be done to accommodate peak time. 9 Document to ACL ratio should be a high value, such as 100:1. 10 Consider data partitioning for large repositories, against the largest Documentum tables, such as dm_sysobject_s and _r. A maxed out cpu and/or memory will cause fluctuating performance levels. This can affect jobs, login times, workflows, etc. Too many ACL's can cause performance issues. Improperly designed security can lead to millions of ACL s. This has a huge performance impact when determining access to documents and folders. This is a factor of database design best practices for both performance and backup/restore reasons. CPU and memory are set for average loads and not for peak load times. Object-level security is set, versus folder or user level, which means at least one ACL for each document, thereby creating a huge excess of ACL s. DBA s don t always know where the cut-off point is for when to partition, or exactly which tables. Documentum Performance Tips Reveille Software Page 4
Longer Term Success: Monitoring Documentum for Continuous Improvement It is important to understand your performance before making any of these modifications, and to have on-going performance monitoring. Service Level Baselines: On-going monitoring creates service level baselines so that you can measure your actual application performance and your actual end-user experience before you change your system. These baselines also are valuable for understanding the impact of upgrades, changes, and expansion of your system. Proactive Management: Monitoring at the application layer allows you to be proactive, versus reactive, when changes are detected in performance and application behavior. Objective Trending and Reporting: In terms of continuous improvement, monitoring allows you to trend your results and continue to make these performance improvement as you change, expand, virtualize, or add applications to your Documentum platform. Through proper configuration management and comprehensive Documentum monitoring, you can optimize service levels and ensure a path of continuous improvement. Reveille Management Console for Documentum The Reveille Management Console for Documentum is the only EMC Certified solution for monitoring actual user experience, application processes and components for Documentum. This agentless, easy-to-implement solution ensures peak application availability and performance of your business-critical Documentum applications. Reveille provides capacity and usage metrics to determine impact of virtualization, hardware and software upgrades, and Documentum platform upgrades. Reveille is scalable to handle the largest Documentum implementations, and lowers support costs and reduces trouble tickets. Documentum Performance Tips Reveille Software Page 5
Reveille Management Console for Documentum includes: Active Documentum Platform Monitoring: Provides for active monitoring of critical Documentum components and processes. Reveille monitors each of the key components, assists in isolating the problem when there is an issue, and can automatically or interactively remediate the issue. Comprehensive Documentum Platform coverage: Provides unique visibility and KPI information for Documentum platform components such as: ACS operation and message failures/success JMS Operation and performance Content Server total/active sessions and job completion DFC/DFS interface performance Virtual and physical Webtop/TaskSpace/D2 (or custom WDK based app) application status Webtop/TaskSpace/D2 (or custom WDK based app) user transaction activity and performance Documentum Performance Tips Reveille Software Page 6
Individual Repository checks within HA environment Actual User Experience Monitoring: Provides you with Documentum application transaction detail to keep your Documentum Webtop / TaskSpace / DFS / D2 based users productive by monitoring the actual user response times and application activity for all users. Supports Secure Sockets Layer (SSL) based traffic. Summary A trial and error approach to optimize Documentum service levels is neither efficient nor cost effective, while performance audits following industry best practices should be utilized periodically to account for complex and dynamic environments that often change to meet demanding business challenges. Effective production monitoring of Documentum is not only necessary, but can be accomplished with the right solution and set the stage for continuous improvements in overall application and resource productivity. Documentum Performance Tips Reveille Software Page 7
Reveille Management Consoles are being utilized to support content management solutions in hundreds of large enterprise deployments, including the top 6 financial institutions in the world. Documentum platform and user activity monitoring across each major computing environment (such as system test, QA, and production) guarantees high service levels and the ability to see and resolve issues before end users are impacted. For more information, download the Reveille Management Console for Documentum datasheet here. About Reveille Software Reveille Software is the provider of the most widely used experience and performance management (EPM) solution for enterprise content management (ECM) applications. Leveraging patent pending technology in use by hundreds of companies including Amgen, Nestle and JP Morgan Chase, Reveille EPM helps companies ensure business-critical applications perform at peak efficiency and availability without application babysitting. Since 2002, Reveille EPM proactively monitors user experience, application processes and business metrics to diagnose and repair failures before end-users experience delay. Reveille Software 3820 Mansell Road, Suite T-20 Alpharetta, GA 30022 t. +1 877.897.2579 / f. +1 770.642.2726 reveille.sales@reveillesoftware.com www.reveillesoftware.com Copyright 2012 Reveille Software. All rights reserved. All other trademarks used herein are the property of their respective owners. Documentum Performance Tips Reveille Software Page 8