NetIQ AppManager Enterprise Scalability FAQs

Transcription

1 NetIQ AppManager Enterprise Scalability FAQs NetIQ AppManager Tiger Team Jeff Chase Andy Doran Neil Harrison Scott Hart Cathal Kennedy Dave Zucker March 2011

2 THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT ARE FURNISHED UNDER AND ARE SUBJECT TO THE TERMS OF A LICENSE AGREEMENT OR A NON-DISCLOSURE AGREEMENT. EXCEPT AS EXPRESSLY SET FORTH IN SUCH LICENSE AGREEMENT OR NON-DISCLOSURE AGREEMENT, NETIQ COR- PORATION PROVIDES THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT "AS IS" WITH- OUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. SOME STATES DO NOT ALLOW DISCLAIMERS OF EXPRESS OR IMPLIED WARRANTIES IN CERTAIN TRANSACTIONS; THEREFORE, THIS STATEMENT MAY NOT APPLY TO YOU. This document and the software described in this document may not be lent, sold, or given away without the prior written permission of NetIQ Corporation, except as otherwise permitted by law. Except as expressly set forth in such license agreement or non-disclosure agreement, no part of this document or the software described in this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, or otherwise, without the prior written consent of NetIQ Corporation. Some companies, names, and data in this document are used for illustration purposes and may not represent real companies, individuals, or data. This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time NetIQ Corporation. All rights reserved. U.S. Government Restricted Rights: If the software and documentation are being acquired by or on behalf of the U.S. Government or by a U.S. Government prime contractor or subcontractor (at any tier), in accordance with 48 C.F.R (for Department of Defense (DOD) acquisitions) and 48 C.F.R and (for non-dod acquisitions), the government s rights in the software and documentation, including its rights to use, modify, reproduce, release, perform, display or disclose the software or documentation, will be subject in all respects to the commercial license rights and restrictions provided in the license agreement. Check Point, FireWall-1, VPN-1, Provider-1, and SiteManager-1 are trademarks or registered trademarks of Check Point Software Technologies Ltd. ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the USA. All other company and product names mentioned are used only for identification purposes and may be trademarks or registered trademarks of their respective companies. For purposes of clarity, any module, adapter or other similar material ("Module") is licensed under the terms and conditions of the End User License Agreement for the applicable version of the NetIQ product or software to which it relates or interoperates with, and by accessing, copying or using a Module you agree to be bound by such terms. If you do not agree to the terms of the End User License Agreement you are not authorized to use, access or copy a Module and you must destroy all copies of the Module and contact NetIQ for further instructions. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQS

3 Table of Contents NetIQ AppManager Version 7.x Site Architecture, Configuration and Sizing... 3 How many agents or objects can be realistically monitored by a single NetIQ AppManager version 7.x site?3 What do I need to know about the NetIQ AppManager repository-based collected data streams and reported events?... 3 When should I consider adding an additional NetIQ AppManager repository?... 4 Can you share any configuration guidelines when setting up a Microsoft SQL server with multiple NetIQ AppManager version 7.x repository instances?... 4 Is this SQL version independent, i.e. is the information below valid for SQL2005/2008/2008 R2 (32 bit and 64bit)?... 5 Does a fault tolerant Microsoft clustered SQL Server/multiple NetIQ AppManager version 7.x QDB repositories system configuration address any scalability limitations?... 5 What are the hardware recommendations, software requirements, and communication ports for the different NetIQ AppManager version 7.x site components (repository, Management Server and Control Center)?... 5 What Communication Ports are used by the NetIQ AppManager Suite?... 8 What are the known performance bottlenecks in a typical NetIQ AppManager version 7.x site? And are there any specific scalability limitations of these various NetIQ AppManager version 7.x core components? Discovered Objects Data Events Jobs Management Servers What scalability and performance guidelines can you provide for virtualized NetIQ AppManager version 7.x core component environments? Does the soon-to-be-released NetIQ AppManager version 8 address any of these known NetIQ AppManager site scalability issues? NetIQ AppManager Version 7.x Management Server Sizing and Scalability Can NetIQ AppManager version 7.x QDB repository capacity be expanded by adding Management Servers? Should I dedicate separate NetIQ AppManager version 7.x Management Servers for Unix-only or Microsoft Windows-only agent communications? Would adding NetIQ AppManager version 7.x Management Servers to an existing site allow me to monitor more deployed agents or objects? What are the best practices for NetIQ AppManager Management Server redundancy? Is there a maximum number of NetIQ AppManager version 7.x Management Servers I can have configured in a site? Are there any Microsoft Windows registry changes which can be applied to my deployed NetIQ AppManager version 7.x Management Servers to improve their scalability and efficiency? How do I handle/address gaps in my NetIQ AppManager version 7.x Knowledge Script collected data and what are the causes and implications of this condition? NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 1

4 NetIQ AppManager Version 7.x QDB Repository and NQCCDB Sizing When should I deploy the Microsoft SQL Server Standard or Enterprise edition in my NetIQ AppManager version 7.x core back-end infrastructure? Are there any best practices for tuning Microsoft SQL servers? With the recent Managed Availability (MA) release of the NetIQ AppManager version 7.x Platform Update, are there any scalability or performance improvements if I decide to upgrade to Microsoft Windows 2008 R2 and Microsoft SQL Server 2008 R2 configurations? How much data should I keep in the NetIQ AppManager version 7.x QDB repository for optimum performance, particularly when the NetIQ AppManager Performace Profiler and NetIQ Analysis Center reporting solutions will be used? NetIQ AppManager Version 7.x Control Center Sizing/Configuration Is there a special case when the NetIQ AppManager version 7.x Control Center CQS component should be separated from the NQCCDB database component? Is the Gold QDB still being implemented? What benefits does this Gold QDB configuration offer in an enterprise-level NetIQ AppManager version 7.x implementation? Are there any NetIQ AppManager version 7.x Control Center component co-location best practices you can recommend? For example, should the NetIQ AppManager version 7.x QDB repository and the NQCCDB along with the CQS be housed on the same Microsoft SQL server? Are there any NetIQ AppManager version 7.x site sizing or configuration recommendations when enterpriselevel Control Center Service Map View (SMV) Console Use is planned? NetIQ AppManager Performance Profiler Sizing Does the NetIQ AppManager version 7.x site scalability get impacted when the NetIQ AppManager Performance Profiler component is added? What scalability best practices guidelines should be followed when the NetIQ AppManager Performance Profiler component is being considered? Can you provide recommended configuration and sizing details for an optimally tuned NetIQ AppManager Performance Profiler system? NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 2

5 NetIQ AppManager Version 7.x Site Architecture, Configuration and Sizing How many agents or objects can be realistically monitored by a single NetIQ AppManager version 7.x site? There is no specific number of monitored agents at which NetIQ AppManager will fail. This makes gauging your implementation difficult, but there are some guidelines that will help explained throughout this document. What do I need to know about the NetIQ AppManager repository-based collected data streams and reported events? As a general guideline, NetIQ AppManager can support 600 to 800 agents per repository. The main issue with scalability is the amount of data collected by the agents. There are a few notable situations that will reduce the number of agents supported by a single repository. Some modules along with NetIQ AppManager Performance Profiler - will impact your NetIQ AppManager infrastructure as if they were more than one agent. o NetIQ AppManager Performance Profiler collects large amounts of data. Just the Microsoft Windows OS template collects 18 data points every five minutes. With NetIQ AppManager Performance Profiler deployed, scalability may be reduced to as low as 400 agents. This is due to the amount of data NetIQ AppManager Performance Profiler collects. For example, the base Windows/NetIQ AppManager Performance Profiler template collects 18 data streams itself. The Microsoft SQL/NetIQ AppManager Performance Profiler template collects data for every SQL database on the server, as well as data per instance for multiple instance SQL installations. You can quickly see how running a few NetIQ AppManager Performance Profiler templates will return an amount of data equivalent to several hundred single agents. Another module that can heavily impact scalability is VMware. This module uses proxy-based monitoring. If you are going to use this module to monitor ESX hosts as well as all of the resources being consumed by the guest virtual machines (VMs), you will need to count every virtual server in the environment as the equivalent of an agent. Typically, every ten minutes the monitoring proxy updates the inventory of the virtual environment and updates the repository. That means that if you are monitoring 500 virtual servers in the environment, you are impacting the repository every ten minutes, as if you had 500 agents deployed. While you can see from these examples that there is no hard figure to use as a limit to scalability, there are things you can monitor that will indicate when the NetIQ AppManager infrastructure has reached its limit. The key is to monitor the persistent input/output cache (PIOC) queues on the Management Servers (MSs). There are performance counters you can monitor, and there are Knowledge Scripts available to help. The other critical component to monitor is the repository. The repository is a high transaction database. Configuring the database server for maximum performance will ensure the greatest scalability available. See the guidelines for optimum configuration starting on page 6 for more information. If you have a complex situation, the best step is to engage your NetIQ field Sales Engineer or NetIQ Professional Services Engagement Manager to assist you with a technical review. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 3

6 When should I consider adding an additional NetIQ AppManager repository? The need for additional repositories can be performance based, business based, or planning based. Need based on performance When an environment reaches a point where either the Management Servers (MSs) or the repository is overburdened, it is time to deploy another repository. Some key metrics to watch include the MS PIOC queue, the NetIQ AppManager repository disk queue length, and the SQL performance (cache hit ratio, blocking, deadlocks, etc.). For more information on these metrics, review Component Choke Points information in this document (see performance bottlenecks, page 13.). Need based on business Any number of scenarios may drive a business-based reason for adding additional repositories, but typically, it is a unique management requirement. For instance, regional deployment scenarios for high availability requirements allow an organization to avoid the single point of failure. In this case, if one site goes offline, monitoring is still available across the remaining sites. With NetIQ AppManager Control Center (Control Center), any commands will be queued until the down repository comes back online. Another business reason may be when there are multiple groups that each has their own monitoring requirements. While Control Center will still provide you a unified view, having separate NetIQ AppManager repositories may be simpler to manage. Need based on planning Often repositories are proactively deployed to accommodate planned changes. The most common changes are mergers and divestitures. In these cases, segregating agents to unique repositories can make it easier to move large groups of monitored agents from one organization to another. Monitoring a large virtual infrastructure may also be a driver for adding an additional NetIQ AppManager repository. -If you have 1500 virtual servers in an environment, the impact of the NetIQ AppManager for VMware module itself could warrant a dedicated NetIQ AppManager repository. As with other repositories, a dedicated VMware repository can be tied via Control Center for a single, IT management view, and you can still report on the repository with NetIQ Analysis Center. Can you share any configuration guidelines when setting up a Microsoft SQL server with multiple NetIQ AppManager version 7.x repository instances? With modern server equipment, it is certainly possible to leverage multiple repositories on a single server. This can take the form of multiple repositories in a single instance or multiple Microsoft SQL server instances. Some things to consider before selecting the best approach for your environment may include: Upgrading your servers. There is only one install path for the server. Any module updates or component hot fixes that make changes to the underlying files (Knowledge Scripts for example) will affect all repositories immediately. You must run the update once for each installed repository, and you must update all installed repositories during the same upgrade session. NetIQ AppManager is a high transaction database. NetIQ recommends that you follow Microsoft s recommendations for optimal performance for high transaction database design. Some key tenants of this are separating database and log files onto separate physical drives or LUNs and TempDBs distributed over multiple processors and disks. Understand your business requirements. Having multiple installations on a single server can be a single point of failure for your infrastructure. Each separate repository will require a dedicated Management Server. This can cause a high I/O load on a single server hosting multiple repositories. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 4

7 Disk drive configuration can be very complex when hosting multiple NetIQ AppManager repositories. Remember that each repository will need separate LUNs or physical drives. See Exhibit 1. Exhibit 1: An example of a multiple repository installation drive configuration. Hard Drives Separate physical drives SAN attached in the RAID configuration listed below will provide the best performance. Separate Microsoft SQL Instances for each repository: RAID 1 OS 100GB Application files (100GB) RAID 10/SAN attached 8 discreet LUNs, 64K cluster formatted drives o Instance 1: LUN 1 TempDB (69GB) 64k LUN 2 NetIQ AppManager QDB1 data files (138GB) LUN 3 NetIQ AppManager QDB1 log files (69GB) o Instance 2: LUN 4 TempDB (69GB) 64k LUN 5 NetIQ AppManager QDB2 data files (138GB) LUN 6 NetIQ AppManager QDB2 log file (69GB) LUN 7 Control Center NQCCDB data/log files (69GB) Is this SQL version independent, i.e. is the information below valid for SQL2005/2008/2008 R2 (32-bit and 64-bit)? The information is not dependent on a specific version of Microsoft SQL. This document is current as of the Microsoft SQL 2008 R2 release. Does a fault tolerant Microsoft clustered SQL Server/multiple NetIQ AppManager version 7.x QDB repositories system configuration address any scalability limitations? A cluster with multiple SQL instances will allow you to distribute those instances to each of the nodes of the cluster. While this makes better use of the available hardware, the occurrence of a failover of one of the instances could cause severe performance issues on the remaining node. For this reason an active-active clustering solution does not really address individual repository scalability. The cluster must be scaled out to the limit that an individual node could feasibly handle the load of all repositories. What are the hardware recommendations, software requirements, and communication ports for the different NetIQ AppManager version 7.x site components (repository, Management Server and Control Center)? Tables 1 and 2 below outline the standard recommended configurations for components for the NetIQ AppManager repository, Control Center server, and Management Server, as well as communication ports used by NetIQ AppManager Suite. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 5

8 Note: It is important to note that the following recommendations are for optimal performance in most cases. An individual implementation, particularly small or large enterprise deployments, should be vetted with NetIQ Professional Services as there may be more options or constraints for system configurations. In general, more data collection will require more robust performance design from the disk subsystem. In general, RAID 5 should be avoided for any high transaction SQL database, such as NetIQ AppManager repositories, because they create write overhead due to parity calculation. Table 1: NetIQ AppManager Repository and Control Center Server Component Recommended Configuration CPU Recommended: 2 x multi-core processors Memory Recommended: 8GB Best performance will be achieved by SAN attached servers using RAID 10. Additional performance will be achieved by segregating the TempDB onto its own spindles. See Microsoft recommendations for high transaction SQL databases or contact NetIQ Professional Services for more information. SAN attached o RAID 1 OS and program binaries o RAID 10 3 LUNs LUN 1 NetIQ AppManager repository data files and SQL core Hard Drives databases LUN 2 NetIQ AppManager repository log files and SQL core logs LUN 3 Control Center data and log files Stand-alone server 4 physical drives o RAID 1 OS and program binaries (possibly TempDB) o RAID 1 Repository data and SQL core databases o RAID 1 Repository log files and SQL core log files o RAID 1 Control Center data and log files Repository data size: Allow 150GB of disk space (A well-maintained repository should not exceed 100GB.). Repository log size: Disk Space Allow 30GB of disk space. NetIQ recommends Simple Mode for SQL so the log size will grow and shrink with the daily operations. NQCCDB database Allow 30GB per repository database NQCCDB log files Allow 6GB per repository database Windows 2003 R2 32 bit current service pack (64 bit supported via Fat Tire managed availability release only.) OS Windows 2008 R2 32 bit/64 bit with current service pack. (NetIQ AppManager 7 managed availability currency pack required.) NIC 1 X gigabit fiber recommended NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 6

9 Component Software Requirements for Each Component Recommended Configuration For NetIQ AppManager repository: SQL 2005 Standard with current service pack. o Note: SQL 2005 Standard limited to 4 physical CPUs but will support multi-core installations ODBC SQL Server driver (SQLSRV32.DLL) and DBNMPNTW.DLL installed in the system directory MDAC 2.6 or higher Account access and password for the SQL Server as login account For Control Center NQCCDB: Microsoft.NET Framework 3.0 or later Microsoft Network DTC service Web Management Server Microsoft IIS 6.0 (IIS 7 configured for 6.0 compatibility mode) o Active Server Pages (ASP) installed and enabled If you are using Windows 2003 and IIS 6.0, you must allow Active Server Page extensions on the computer where you plan to install the Web Management server. o Active Data Object (ADO) installed and enabled o MDAC 2.6 or higher o ODBC SQL Server driver (SQLSRV32.DLL) and DBNMPNTW.DLL installed in the system directory o Microsoft XML Parser version 3.0 (msxml3) SP1 or later Control Center Command Queue Service Service account that is a local administrator and NetIQ AppManager administrator for each NetIQ AppManager repository Microsoft.NET Framework 3.0 or later Deployment Service Service account that is a local administrator Microsoft.NET Framework 3.0 or later Microsoft Background Intelligent Transfer Service (BITS) Client Component version 1.5 or later Optional SSL Certificate installed. Allows the Deployment Service to run in proxy mode to access the Deployment Web Service across a firewall. Deployment Web Service Microsoft.NET Framework 3.0 or later Microsoft IIS with the following components enabled o ASP.NET o BITS Server Extensions o Default IIS Web Site o Default port bindings (80 for HTTP and 443 for secure connections) Optional - SSL Certificate installed. Allows the Deployment Service to run in proxy mode to access the Deployment Web Service across a firewall NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 7

10 Table 2: Management Servers* Component CPU Memory Hard Drives OS NIC Recommended Configuration Dual processor current type 2-4GB OS drive no significant data storage on this component Windows 2003 R2 32 bit current service pack Windows 2008 R2 32 bit/64 bit with current service pack. (NetIQ AppManager 7 managed availability currency pack required.) 1 X gigabit fiber recommended ODBC SQL Server driver (SQLSRV32.DLL) and DBNMPNTW.DLL installed in the system directory Software Requirements MDAC 2.6 or higher Availability of TCP port 9999 Microsoft XML Parser 6.0 SP1 or later (msxml6) *It is a NetIQ best practice to have two Management Servers for failover and load balancing. What Communication Ports are used by the NetIQ AppManager Suite? Product Communication Protocol Port Operator Console to Repository (QDB) SQL ODBC 1433 Management Server (MS) to Repository (QDB) SQL ODBC 1433 Web Management Server (WMS) to Repository (QDB) SQL ODBC 1433 Control Center Console to Control Center Repository (NQCCDB) SQL ODBC 1433 Control Center Command Queue Service to Control Center Repository (NQCCDB) SQL ODBC 1433 Repository (QDB) to Operator Console Agents to Deployment Web Service Deployment Service running in proxy mode to Deployment Web Service SQL ODBC TCP/IP UDP/IP 135 * Web Operator Console to WMS (standard) Agents to Deployment Web Service Deployment Service running in proxy mode to Deployment Web Service HTTP 80 Web Operator Console to WMS (optional) HTTPS 443 Deployment Service running in proxy mode to Deployment Web Service HTTPS 443 Management Server (MS) to Windows Agents (MC) TCP/IP 9998 Windows Agents (MC) to Management Server (MS) TCP/IP 9999 UNIX Agents (MC) to Management Server (MS) TCP/IP 9001 NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 8

11 Product Communication Protocol Port Deployment Service to target computer for installation TCP/IP 139 Troubleshooter and NetIQCtrl (Operator Console) to/from Agent TCP/IP 8996 * NetIQ ResponseTime for Networks proxy to Performance Endpoints NetIQ Vivinet Assessor to Performance Endpoint 1 for test setup NetIQ Vivinet Diagnostics to Performance EndPoint 1 for test setup Performance Endpoint 1 to Performance Endpoint 2 for test setup Performance Endpoints to NetIQ Response Time for Networks proxy (results) Performance Endpoint 2 to Performance Endpoint1 (report results) Performance Endpoint 1 to NetIQ Vivinet Assessor (report results) Performance Endpoint 1 to NetIQ Vivinet Diagnostics (report results) TCP/IP * TCP/IP Performance Endpoint to Performance Endpoint (VoIP calls ) UDP/IP Auto range * # SNMP Queries UDP/IP 161 SNMP Traps (Source to Receiver) UDP/IP 162 NetIQ Vivinet Diagnostics to Avaya CS1000 Call Servers (rlogin) TCP/IP 513 NetIQ Vivinet Diagnostics to Avaya CS1000 Signaling Servers (Telnet) TCP/IP 23 Avaya Communications Server proxy server to Avaya CS1000 devices on ELAN (FTP to collect OMReports) Avaya CS1000 devices on ELAN to Avaya Communications Server proxy server (FTP data to collect OMReports) TCP/IP 21 TCP/IP 20 Cisco Call Manager to proxy server (FTP of call records) TCP/IP 21 Cisco Call Manager to proxy server (FTP data for call records) TCP/IP 20 Cisco AXL TCP/IP 8443 * Indicates a bidirectional port requirement. Indicates additional port requirements. If you are using a named instance of SQL Server, or if you are not using the default SQL Server port (1433), additional port requirements include: The SQL Server Browser port, The SQL Server Browser service helps clients determine the associated SQL Server port to use. Once a client establishes a connection to the SQL Server running on the non-default port, it will not use the SQL Browser again unless the SQL Server port changes. The SQL Server port for the instance that is hosting your NetIQ AppManager repository and Control Center repository. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 9

12 Indicates that an additional port range is needed. If you plan to remotely install agents and updates across a firewall, you need to decide how many ports you want to allocate to DCOM processes on the agents. # Indicates auto range of ports for Voice over IP (VoIP) test calls. Unless manually specified, ports used are even numbered ports in range to Ports can be configured for test calls going through a Indicates default value. Can be changed, and must be unique for each monitored Avaya Call Server. What are the known performance bottlenecks in a typical NetIQ AppManager version 7.x site? And are there any specific scalability limitations of these various NetIQ AppManager version 7.x core components? Discovered Objects The number of objects discovered in NetIQ AppManager directly affects the performance of the Operator Console. The Operator Console is a fat client front end to the repository. This means that it is constantly refreshing every object it displays and the number of agents discovered can exceed 1,000 without much performance impact. However, each object includes a number of child objects under those agents that can negatively impact the Operator Console, to the point that it fails to load. Each discovered server has child objects based upon subsequent discoveries such as hardware and applications (Microsoft SQL, Microsoft Exchange, Microsoft Active Directory, etc.). As the number of total objects increases, the amount of physical memory the Operator Console process (netiq.exe) consumes also increases. For an environment in excess of 500 agents, it is strongly recommended that the Operator Console be run only on machines with 1GB or more of available physical RAM. Certain applications that discover a large number of sub-objects need to be considered in the equation as well, such as VMware, NetworkDevices, or print servers with a very large number of queues present. The number of objects will also affect the number of data streams created by monitoring jobs, assuming data is collected for each object. For example, a Microsoft SQL server with five application databases will generate child objects for the core SQL databases and each application database. When you run a job against this server, you can potentially collect data for each of the databases. This in turn can negatively impact the Operator Console s Graph Data view, the Chart Console, and the Report knowledge scripts. Control Center, on the other hand, is not directly affected by the number of objects present. Data The amount of data collected and retained within NetIQ AppManager directly affects the size of the database. This, in turn, affects the processes responsible for reporting on or grooming that data. The Chart Console, Report Knowledge Scripts, and stored procedures will take longer to complete as well as the operations used to purge older data. This can cause database blocking, which prevents new data from being inserted by the Management Servers in a timely fashion. This slow performance will also lead to overall performance issues from deadlocking (processes being terminated by SQL in an attempt to preserve integrity) and Management Servers filling their PIOC queues while trying to insert data. Slow Management Servers can then lead to further complications by delaying Event insertion; causing problems with NetIQ AppManager Performance Profiler because data will not be available for processing in a timely fashion; and Management Server invoked actions can fail due to time out or lack of resources available. For these reasons, it is important to implement a data collection and cleaning strategy when deploying NetIQ AppManager rather than waiting until the database becomes too large. In addition, if data reporting applications such as NetIQ AppManager Performance Profiler or NetIQ Analysis Center are being used, there may be little value in keeping data in the repository as it is copied to each of these components separately. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 10

13 Events The number of events present in a NetIQ AppManager repository directly affects the performance of the Operator Console. Regardless of the status of events (Open, Closed, Deleted), the number of events present causes the Operator Console to run incrementally more cycles as it attempts to filter the results. In addition, a large number of events make it impossible to separate critical issues from non-actionable noise (white noise). Therefore, it is important to set up effective event grooming properties within NetIQ AppManager, as well as to define the alerts in each job so that only critical, actionable issues are reported. There are settings for Events handling in the Operator Console. (These are located under File, Preferences, the Repository tab, and the Event button.) These settings can automate event closure and deletion and table grooming. There are also custom stored procedures, such as Auto Age Events, that will help automate this required maintenance. Many of these customizations are available on the Knowledge Depot or via NetIQ Sales Engineers and consultants. Jobs While there really is no choke point directly attributable to the number of jobs present in an environment, the following items need to be considered prior to deployment: There is no physical limit to the number of jobs that can run on a given agent; however, the type of jobs run will have an effect on the number of jobs that can be run. Some Knowledge Scripts take a significant amount of time and system resources to collect desired information, or may collect a large volume of information in a single iteration. Care must be taken when deploying such jobs. It is recommended that you monitor job performance and behavior by looking at Job Detail on an agent via Troubleshooter in the Operator Console or NetIQCtrl using the Profile command. This will provide the average iteration time for each job, and will indicate if any jobs are taking a long time to complete and possibly causing the failure of other jobs. It is important to consider the amount of data that will be returned by Knowledge Scripts. For example, the VirtualCenter_VM* Knowledge Scripts will create one or more data headers and points for each virtual machine that is monitored. For a server that has several hundred VM objects, this can mean thousands of data points coming in from a single Knowledge Script. In extreme cases, this can take several minutes to complete. It is therefore very important to first consider if the information collected is absolutely necessary. If NetIQ AppManager Performance Profiler is involved, it is important to account for a large volume of data that will be generated from each NetIQ AppManager Performance Profiler template (Analytics Knowledge Script). Management Servers In a busy implementation, the volume of information coming into the Management Server at some point is going to overwhelm the process. The PIOC files are designed to alleviate this situation. However, this becomes a performance bottleneck when there is a constant rising PIOC queue count. An overall tipping point is present as the overall rate of incoming data exceeds the rate at which the data can be inserted into the NetIQ AppManager repository database (QDB). It is important to watch for this condition as it can lead to lost data and overall performance degradation. At the point where the Management Servers are no longer able to handle the load of incoming data, the Management Servers begin to block each other from inserting data. The only real solution for managing the increase in data is an additional repository. Another Management Server impact is MS-based actions. The system installation default is that all actions are performed by the Management Servers. In implementations that are larger or have many actions, this can lead to overloading of MS resources, which can in turn cause the agent services to become unresponsive. This situation can be mitigated by configuring actions to be agent based or to utilize a designated proxy agent. What scalability and performance guidelines can you provide for virtualized NetIQ AppManager version 7.x core component environments? NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 11

14 NetIQ AppManager works equally well on virtual or physical servers as long as Microsoft SQL Server and NetIQ performance recommendations are followed. Many SQL DBAs have individual preferences regarding virtual databases. If the virtual servers provide equivalent performance to your hardware recommendations, there is no reason you cannot virtualize any component of NetIQ AppManager. Does the soon-to-be-released NetIQ AppManager version 8 address any of these known NetIQ AppManager site scalability issues? No, NetIQ AppManager version 8 does not address the scalability guidelines in this document. It focuses on usability, enhanced functionality, and Control Center performance. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 12

15 NetIQ AppManager Version 7.x Management Server Sizing and Scalability Can NetIQ AppManager version 7.x QDB repository capacity be expanded by adding Management Servers? No. Adding Management Servers can help with certain performance and data management issues, but in general, it will not affect repository scalability. If the scalability limitation is due to an overburdened repository, adding a Management Server will actually exacerbate the problem. If the scalability problem is due to a particular agent collecting a lot of data, such as a dedicated VMware agent, it may help to unburden the other Management Servers and improve overall performance. Should I dedicate separate NetIQ AppManager version 7.x Management Servers for Unix-only or Microsoft Windows-only agent communications? If you are using the current version of the Unix agent and the Management Server, there is no technical reason to do this. However, you may still have a valid business reason for wanting to segregate, such as working with a DMZ environment. Would adding NetIQ AppManager version 7.x Management Servers to an existing site allow me to monitor more deployed agents or objects? No, generally adding Management Servers does not affect the scalability of the overall deployment. In some cases involving small deployments (with all core components on one server), adding a Management Server may impact scalability. Often times in these cases, simply moving the existing Management Server to a dedicated server will achieve the same effect. Scalability is not achieved by adding additional Management Servers but through adding additional repositories. What are the best practices for NetIQ AppManager Management Server redundancy? NetIQ recommends using multiple Management Servers. It is possible to cluster the Management Server using Microsoft Cluster Server, but this is only supported in an Active/Passive configuration. This configuration causes you to essentially keep one server idle for every working Management Server. A more efficient solution is utilizing multiple Management Servers. This allows you the added benefit of redundancy plus load balancing the workload from your agents. The key to this design is to make sure that, in the case of any single Management Server failure; the remaining Management Servers can handle the existing agent load. Is there a maximum number of NetIQ AppManager version 7.x Management Servers I can have configured in a site? Best Practice recommendation NetIQ does not recommend more than eight Management Servers in a single NetIQ AppManager site. Technically, there is no physical or hard coded limit. There is no linear relationship between the number of Management Servers and the amount of work the servers can handle from agents. For example, if one Management Server can process 1,000 data points in one second, it does not follow that two Management Servers could process 1,000 data points in half a second. As more Management Servers are added, they will eventually contend for resources in the NetIQ AppManager repository. In most cases you should have at least two Management Servers. As you surpass four Management Servers, you will likely see a performance impact in NetIQ AppManager. This is because each Management Server accesses the repository every five seconds. The Management Server both queries for any updates to its managed agents, as well as lock the various NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 13

16 database tables to insert new information. The more Management Servers performing these actions on a single repository, the greater the chance the servers will block each other s access to the database. Are there any Microsoft Windows registry changes which can be applied to my deployed NetIQ AppManager version 7.x Management Servers to improve their scalability and efficiency? There are a few registry keys that can impact the performance of your Management Servers. Under the HKLM\Software\NetIQ\AppManager\4.0\Netiqms\Config key, check or make the following recommended values changes: Value PIOC Data Map File Size MB PIOC Event Map File Size MB PIOC JobStat Map File Size MB Set Value or Path to: Persistent IOC Make sure this is set to 1 PIOC Map File Path Data Thread Set to 20 (This is the new default for the most current NetIQ AppManager 7 release.) Set to 20 (This is the new default for the most current NetIQ AppManager 7 release.) Set to 20 (This is the new default for the most current NetIQ AppManager 7 release.) Ensure that this path is valid or the NetIQ AppManager Management Server can fail Make sure this is set to the default 2 value Under the HKLM\Software\NetIQ\AppManager\4.0\Netiqms\Tracing key, check or make the following recommended value changes, which control the amount of logging the Management Server performs. The following settings restore the Management Server to default minimums and will reduce the workload on a busy one: Value Set this to: Action Log Set this to 0 QDB Log Set this to 0 RPC Log Set this to 0 Tracedata Set this to 0 TraceXML Set this to 0 If the Management Server is working with a large amount of Unix agents, the following registry changes can improve the performance in a busy NetIQ AppManager deployment. Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters key, add or edit the following listed values: TcpTimedWaitDelay=dword: e MaxUserPort=dword: KeepAliveInterval=dword: NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 14

17 How do I handle/address gaps in my NetIQ AppManager version 7.x Knowledge Script collected data and what are the causes and implications of this condition? Data gaps can be caused by a myriad of issues. Some of the most common include: Stopped jobs Failed jobs Failed agents Agent performance problems Job overruns when a job iteration takes longer than the job s scheduled interval Management Server problems In all cases, there is no way to rebuild or recover data gaps; however, these gaps should guide you to validate that a specific agent, job, or Management Server is functioning properly. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 15

18 NetIQ AppManager Version 7.x QDB Repository and NQCCDB Sizing When should I deploy the Microsoft SQL Server Standard or Enterprise edition in my NetIQ AppManager version 7.x core back-end infrastructure? Microsoft SQL Server 2005/2008 Standard Edition will suffice from a performance standpoint. The Enterprise Edition does not offer any specific NetIQ AppManager performance improvement over the Standard Edition. For the full capabilities please reference: Microsoft SQL Server Microsoft SQL Server Explanation of multi-core vs. multiple processor capabilities Are there any best practices for tuning Microsoft SQL servers? Yes, see the list below for best practice recommendations: 1. Use the fastest disk subsystem you can afford. SQL is an I/O sensitive application. 2. Allocate as much memory as you can for SQL leave 1GB for the operating system and set the minimum/maximum memory usage for SQL to be all the remaining physical memory (so if you have 4GB, allocate 3GB to SQL). This is based on the assumption that SQL is the only application on the system. 3. Always configure the pagefile(s) to be away from the system disk and of a fixed size (growing a pagefile dynamically causes performance issues). 4. For data file growth, the default in SQL Server 2005 and 2008 is for it to grow by 1MB. This will cause massive fragmentation and will lead to serious performance degradation. Set this to 10 percent for your data files. A more difficult but performance-tuned approach is to monitor the databases in order to be able to manually adjust them. It is more efficient to grow databases manually than have the system automatically expand them due to the overhead of the calculation and disk write. 5. Ensure the TempDB is configured properly according to Microsoft recommendations. It should be located separately, not on the SQL application disk, which is the default. It also should not be on any disk containing production databases. It should be sized appropriately and this depends upon how big the production database is. It should be at least 1GB for the TempDB and not more than 10GB. A further enhancement is having one TempDB data file per physical CPU up to a maximum of eight data files. With multiple data files, ensure their size and growth rates are all identical. For example, a TempDB that needs to be 4GB on a 4 CPU server means 4 * 1GB data files. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 16

19 With the recent Managed Availability (MA) release of the NetIQ AppManager version 7.x Platform Update, are there any scalability or performance improvements if I decide to upgrade to Microsoft Windows 2008 R2 and Microsoft SQL Server 2008 R2 configurations? No, the NetIQ AppManager version 7.x Platform Update only allows installation on Microsoft Windows 2008 R2 with SQL Server 2008 R2 in either 32- or 64-bit configurations. This can shorten the time for an upgrade to NetIQ AppManager version 8 by allowing the new hardware to currently host NetIQ AppManager version 7.x. The underlying code has not changed and it will improve performance, but not scalability. How much data should I keep in the NetIQ AppManager version 7.x QDB repository for optimum performance, particularly when the NetIQ AppManager Performance Profiler and NetIQ Analysis Center reporting solutions will be used? NetIQ recommends that a NetIQ AppManager repository not exceed 100GB in size. For best performance, keep this to less than 60GB. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 17

20 NetIQ AppManager Version 7.x Control Center Sizing/Configuration Is there a special case when the NetIQ AppManager version 7.x Control Center CQS component should be separated from the NQCCDB database component? No, this is to accommodate the clients that have strict rules governing their SQL Server farms. Many companies do not allow any executables, services, or other non-sql items to be loaded. In these cases, you can move the CQS to a discreet server that does not require a SQL license. Is the Gold QDB still being implemented? Yes, the Gold QDB design still has some use. What benefits does this Gold QDB configuration offer in an enterprise-level NetIQ AppManager version 7.x implementation? The original reason the Gold QDB was created was to address a technical problem for enterprise customers. That problem has been solved with Control Center updates in NetIQ AppManager, version Currently, you would only implement the Gold QDB design for specific business reasons. There is no technical reason for this solution. The reasons include: In a multiple-repository environment, the Gold QDB provides added flexibility to move and change repository servers. This is especially true if several of the servers are about to undergo a hardware refresh. It might be simpler to build a Gold QDB design on a small server. In multiple Active Directory-forest environments, you can build this server in a central, trusted forest to allow for easier administration. This is especially true for mergers and acquisitions or distributed IT control. We strongly recommend that you consult with NetIQ Professional Services before implementing this. It does make the overall solution a little more complex because it adds an additional repository that needs to be maintained. Also, it can increase costs due to additional licensing required for Microsoft SQL Server and Microsoft Windows. The Gold QDB was developed as a field Professional Services solution and there are no official product documents covering its setup and usage. Are there any NetIQ AppManager version 7.x Control Center component co-location best practices you can recommend? For example, should the NetIQ AppManager version 7.x QDB repository and the NQCCDB along with the CQS be housed on the same Microsoft SQL server? The Control Center database is a very low impact database. If the NetIQ AppManager repository is not heavily burdened, there is no reason that Control Center cannot be collocated on the same server. The Command Queue Service can be installed anywhere in the environment, but remember that it is constantly communicating with every NetIQ AppManager repository and the Control Center database. That means that the server hosting this service must be well connected to all the appropriate databases. There may be business reasons to segregate the Command Queue Service. Many DBAs do not allow any executables, services, or other non-sql items to be loaded on the SQL servers. In these cases, you can move the CQS to a discreet server that does not require a SQL license. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 18

21 Are there any NetIQ AppManager version 7.x site sizing or configuration recommendations when enterprise-level Control Center Service Map View (SMV) Console Use is planned? The biggest problem an enterprise faces is creating Service Maps that are nested too many levels deep. This can lead to slow performance. A good rule of thumb is that if a design calls for service maps with more than four layers of nesting, you should reevaluate your design. Often times, several smaller map configurations will suffice and provide a quicker and more flexible solution across a broader set of users. Another rule of thumb is to make sure that the objects used are expected to remain in the environment for a long time. If a discovery fails on an object or an object is deleted, the maps using those objects will have to be recreated because the underlying object identifiers have changed. Leveraging event views and server views will provide for dynamic updates to the service maps and can definitely lessen the administrative overhead. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 19

22 NetIQ AppManager Performance Profiler Sizing Does the NetIQ AppManager version 7.x site scalability get impacted when the NetIQ AppManager Performance Profiler component is added? NetIQ AppManager Performance Profiler will generate a lot of data streams for NetIQ AppManager. You should be well aware of the recommendations in this document for sizing, performance, and tuning. As an example, NetIQ typically figures about 2MB of raw data per server per day with NetIQ AppManager alone. When NetIQ AppManager Performance Profiler is added, this data estimate doubles. In other words, deploying NetIQ AppManager Performance Profiler can have the same impact as doubling the number of agents on your existing NetIQ AppManager infrastructure. What scalability best practices guidelines should be followed when the NetIQ AppManager Performance Profiler component is being considered? Before deploying NetIQ AppManager Performance Profiler to an existing environment, NetIQ recommends contacting your NetIQ Professional Services Engagement Manager for assistance with planning a NetIQ AppManager Performance Profiler implementation. Can you provide recommended configuration and sizing details for an optimally tuned NetIQ AppManager Performance Profiler system? The best resource for this information can be found in the NetIQ AppManager Performance Profiler Getting Started PDF document: Reference pages 4 through 6 for configuration and sizing recommendations. Note: If the NetIQ AppManager Performance Profiler Application Server and Database Server components are going to reside on the same physical machine, simply add the information together to obtain the actual server specifications. Standard database configuration guidelines also apply. NETIQ APPMANAGER ENTERPRISE SCALABILITY FAQs 20