BPPM 9.5 Architecture & Scalability Best Practices 2/20/2014 version 1.4

Transcription

1 Summary This Best Practice document provides an overview of the core BPPM 9.5 architecture. Detailed information and some configuration guidance is included so that the reader can get a solid understanding of the core solution components regarding how they are connected and communicate with each other. Best Practice recommendations are provided throughout. References to previous versions are provided and discussed where appropriate. Caveats & Limitations This document covers basic implementation architecture and does not cover all possible functions. It is focused on the core components of BMC ProactiveNet Performance Manager v9.5. For example BPPM Reporting is not included. BPPM Reporting is addressed separately. The document also does not include all possible implementation architecture options. Although the solution is very flexible and can be implemented in multiple ways, this document follows Best Practice recommendations. The information here is intended to augment the product documentation, not replace it. Additional solution and product information can be found in the product documentation at the following URL. The port numbers provided in this document are based on a default implementation. Page 1 of 47

2 Table of Contents Summary... 1 Caveats & Limitations... 1 BPPM 9.5 Overall Architecture... 4 Architecture changes compared to BPPM BPPM Server Architecture... 6 Sybase Database Architecture... 7 Oracle Database Architecture... 8 Service Hosts Event & Data Flow Processing Connection Details Central Management & Administration (CMA) Single CMA Architecture Overview Multiple CMA Architecture Standalone BPPM Servers & CMA CMA Architecture Details Staging Services Overview & Functionality Staging Process Illustration Initial Agent Deployment Service Policy Application Monitoring Staging & Policy Management for Development, Test and Single CMA Instance Deployments Multiple CMA Instance Deployments General Recommendations Interoperability High Availability BPPM Application Server HA Data Collection Services HA Staging Service HA Event Management Cells HA Page 2 of 47

3 s HA Sybase Database HA Oracle Database HA BPPM 9.5 Scalability & Sizing BPPM Server Sizing Overview Service Node Sizing Overview Configuring for Scalability Implementation Order Components & Dedicated Servers Troubleshooting Page 3 of 47

4 BPPM 9.5 Overall Architecture The diagram below illustrates the high-level architecture of the BPPM 9.5 core solution components. User Consoles (Web GUI & Java Admin) Event Management Correlation Cell BPPM Server Web Operations Console Service Impact Management & Alerting Event Management Root Cause Analytics Monitoring / Trending / Reporting Central Management & Administration Local Sybase Database or Remote Oracle Database (RAC supported) Service Hosts 1) Service 2) Event Management Cell 3) Event Adapters (optional) 4) RT Server (optional) Remote Monitoring Events & Performace Data Events & Performace Data Events & Performace Data Transaction Response Time & 3 rd Party Data Sources Remotely Managed Nodes Locally Managed Nodes s Local Monitoring Legend Key Data & Events Data Events s collect performance data and generate events for availability metrics. Both performance data and events from PATROL are streamed though the Service nodes. (This assumes the BPPM 9.5 Server and BPPM 9.5 Service nodes are in use.) The Service nodes forward the performance data to the BPPM Server. Not all performance data has to be forwarded. Performance data can be collected and stored at the PATROL agents and visualized as trends in the BPPM 9.5 console without having to stream the data to the BPPM Server. This is configurable for each PATROL parameter. It is a Best Practice to limit streaming performance data to the BPPM server for only the following purposes. 1) Performance data for all parameters designated as KPIs should be streamed to the BPPM server to support baselines, abnormality detection and predictive alarming. Page 4 of 47

5 2) Performance reporting in BPPM Reporting. Stream the data for all parameters that are required in performance reports. This should be limited to KPI parameters, but can be extended. 3) Include parameters that are necessary or desired for probable cause analysis leveraging baselines and abnormalities. The Service processes forward events to event management cells running on the Service hosts. The event management cells running on the Service hosts filter and enhance events, then forward the events to an event management cells used for correlation. Best Practices and the options available are discussed further in this document. BMC strongly recommends that you setup environments for BPPM development and BPPM test separate from production. Architecture changes compared to BPPM 9.0 Much of the overall architecture remains unchanged from the previous release; however there are some significant changes. The major high-level changes are listed below. 1) The Service process has been significantly simplified. 2) Support for multiple Oracle schemas in the same Oracle instance is provided. This applies to the BPPM Application Server database and the BPPM Reporting database. 3) Connectivity between the Service processes and the Central Management & Administration module (CMA) has been consolidated. CMA now communicates with each Service through the BPPM Server that the Service is connected to. 4) In 9.0 data is sent from s to the Service nodes, but the BPPM Server polls a single data point every 5 minutes from the Services. In 9.5 data is now streamed from s through the Service to the BPPM Server. Consequently, every data point is now collected by the server and stored in the database for performance parameters that are streamed. 5) With BPPM 9.5 events are now streamed from s to the Services on the same port that performance data streams to. The Service then sends the events to remote cells or directly to the BPPM Server. In BPPM 9.0 s send events directly to remote cells on a separate port. Details regarding these changes are discussed further in this document. Page 5 of 47

6 BPPM Server Architecture The BPPM solution supports installing a single BPPM Server or multiple BPPM Servers in the same environment. The overall architecture diagram on page 1 illustrates a single server environment. The diagram below illustrates a multiple BPPM Server environment with a Central BPPM Server with the Central Management and Administration (CMA) module and Child Servers. Central BPPM Server with CMA Legend Key Policies Data & Events Data Events Modeling Direction of arrows indicates connection requests. Child BPPM Server 1 Child BPPM Server 2 Child BPPM Server N Child BPPM Server N+1 Host 1 Host 2 Host N Host N+1 monitoring monitoring monitoring monitoring A multiple BPPM Server implementation supports distributed service models so that specific Configuration Items in one BPPM Server can be visible in another model in a separate server. This is supported by Web Services installed standard with the BPPM Server(s). The Central BPPM Server acts as a single point of entry for users and provides a common point to access service models. Although not required, for most environments BMC recommends installing the top tier BPPM Server as a Central server with the CMA module included. A single BPPM solution implementation cannot support mixed versions of BPPM servers. This includes the Central Management and Administration module. All BPPM Server versions must be the same in a single environment. Page 6 of 47

7 The following are best practices for the BPPM Server. 1) Install and use the BPPM Administration console on a host separate from the BPPM Server. Use the instance of the BPPM Administration console that is installed with the BPPM Server for emergency use only. 2) Install IIWS and all other integrations on separate servers from the BPPM Server. All integrations should be installed on a server separate from the BPPM Server (for example on an Service host) unless specifically otherwise stated in BMC documentation. This does not apply to the for BMC Remedy Service Desk (IBRSD). 3) Install a and the Monitor the Monitor Knowledge Module (KM) on the BPPM Server in order to monitor the BPPM Server externally. The BPPM Server includes built in self monitoring, however the Monitor the Monitor KM provides a way to monitor the BPPM Server externally. 4) Setup a separate event/notification path for external monitoring of the BPPM infrastructure so that you are not dependant on the BPPM infrastructure to generate and process alarms related to it being down or running in a degraded state. 5) Do not try to forward performance data to a Central BPPM Server. Performance Data cannot be forwarded to a Central BPPM Server. Only events can be forwarded to a Central BPPM Server. Sybase Database Architecture The BPPM Server is supported with one of two database options. You can install the embedded Sybase database that comes with the product, or you can leverage an Oracle database that you provide. If you choose the Sybase option, the Sybase database is installed with the BPPM Server on the same host with the application server and web server components. The Sybase database cannot be installed on a separate server. The Sybase database should be used in the following situations. 1) Oracle License is not available 2) No Oracle DBA is available 3) Robust Database availability is not required 4) Small & medium environments where Oracle is not available Please see the product documentation for details regarding database topics. Page 7 of 47

8 Oracle Database Architecture If you choose the Oracle database option you must provide an Oracle instance. The Oracle instance must be installed on a separate host from the BPPM Server. You have the option of allowing the BPPM Server installer to create the schema for BPPM in the Oracle database, or you can create the schema manually using scripts provided with the installer. Please see product documentation for additional details regarding the install options and process. With BPPM 9.5, multiple BPPM Application Servers can be supported with a single Oracle instance. This is accomplished by creating/allocating separate Oracle schemas in the single Oracle instance, one for each BPPM Application Server. Obviously database resources and the sizing of the Oracle instance SGA have to be increased to support this. The diagram below illustrates how multiple BPPM Servers can share a single Oracle instance. NOTE: This is not possible with the Sybase database. Central BPPM Server with CMA Oracle Server Single DB Instance Development BPPM Server Test BPPM Server BPPM Server N BPPM Server N+1 Likewise, multiple BPPM Reporting instances can share the same Oracle instance. WARNING: An Oracle instance should never contain a schema or schemas for the BPPM Server while also containing a schema or schemas for BPPM Reporting. The BPPM Application Server instance(s) and reporting instance(s) must be separated for performance reasons. Additionally the Oracle database instances for BMC components should be dedicated for BMC products and should not contain any third party application data or code. The diagram below illustrates these requirements. BPPM Server 1 Report Engine 1 BPPM Application Database BPPM Reporting Database Schema 1 BPPM Server N Report Engine N Schema N Reporting Schema Schema N+1 BPPM Server N+1 Report Engine N+1 Page 8 of 47

9 Each schema in an Oracle instance must have a unique Oracle database user that owns the schema. When you install the BPPM Server the installer prompts you for the user who owns the schema for the current instance as shown in the screen below. Be sure to enter a unique user for each BPPM Server instance you install. Additionally, each unique BPPM Server schema should be installed into separate data files and corresponding tablespaces in the Oracle Instance. The BPPM Server installer allows you to specify these criteria as shown below. Page 9 of 47

10 The BPPM Application Server installer requires remote connectivity to the Oracle instance and must be able to connect as sysdba remotely. You should validate this connectivity before trying to install the BPPM Application Server. It is a Best Practice to install SQL*Plus or the Oracle Instant Client on the target BPPM Application Server and test/validate Oracle database connectivity as sysdba from that server before starting the install for the BPPM server. Please see product documentation for additional information regarding Oracle. Oracle can be configured so that each database instance has a unique Oracle listener, or a single listener can support multiple database instances. As a best practice it is recommended to designate a unique Oracle listener for each database instance. This isolates listener issues to a single instance. Additionally high availability should be setup for the Oracle listeners and the databases. BMC recommends leveraging Oracle RAC for database high availability. Please see BMC product documentation for details regarding BPPM and Oracle RAC. Please see Oracle documentation for additional Oracle related high availability configuration. BMC recommends leveraging the same database platform for all BPPM Server databases across the environment. Although it is technically possible to install some BPPM Servers using the embedded Sybase database and others using Oracle, standardizing on one platform provides a common way to manage high availability, backup/restore, and export/import of data from one instance to another. Note that database export/import is only possible between the exact same versions and patch levels. In previous releases of BPPM each instance of both the BPPM Application Server and the BPPM Reporting components required a dedicated Oracle instance. (This assumes Oracle was the chosen database for the BPPM Server, not Sybase.) The Oracle database option should be used in the following situations. 1) Large environments 2) When an Oracle License is already available 3) The customer has on site Oracle DBA expertise 4) Oracle is a standard database platform used in the environment 5) When robust database availability is required The following are additional best practices when using Oracle as the database platform. 1) Use Oracle RDBMS v or v ) Create at least two BMC ProactiveNet users, one for data storage and one for data views. Consider a third backend user to manage issues like locked accounts. 3) Physically co-locate the BPPM App Server and the DB Server on the same subnet. 4) The backup and restore process must be executed by BMC ProactiveNet users. Page 10 of 47

11 5) Use BMC Database Recovery Management or an Oracle tool such as RMAN. 6) Enable archive logging. 7) Use Oracle RAC for High Availability 8) Use Oracle Data Guard for Disaster Recovery 9) Use Oracle Storage Area Network (SAN). Page 11 of 47

12 Service Hosts BPPM 9.5 Architecture & Scalability Best Practices The diagram below illustrates how Service nodes fit into the BPPM 9.5 architecture. A reference to the 9.0 architecture is provided on the left for comparison. BPPM 9.0 All Environments BPPM 9.5 Very Small or POC Environments BPPM 9.5 Recommended BPPM Server BPPM Server BPPM Server Events Data Data & Events Events Data Event Cell Service Service Event Cell Service Service Host Service Host Service Host Events Data Data & Events Data & Events Nodes (version 9.0) Nodes (version 9.5) Nodes (version 9.5) Data & Events Data Events Legend Key Direction of arrows indicates connection requests. The BPPM 9.5 Service processes accept streaming of PATROL data and events using a common connection port. The default port is This includes all data points and events from PATROL for parameters that you select. Once events arrive at the Service, events are separated and follow a unique path to one of the following based on configuration: 1) The Service local cell (default behavior) 2) A Named Event Cell 3) The BPPM Server associated to the Service NOTE: PATROL sends performance data, streaming it to the BPPM server. This is not summarized data. The data does get summarized in the BPPM Server (as in previous versions) but raw data is sent from the s. This includes all data points for parameters that you decide to send. Page 12 of 47

13 The architecture also supports buffering of PATROL performance data and events at the s in case there is a network connectivity issue or the Service otherwise cannot be reached. When the reconnects to an Service process the buffered data is sent. This capability is not intended to support buffering for very large amounts of data. It is intended to support a few minutes of lost connectivity, not hours or days. Testing has shown that the process can support up to 30 minutes of data collected by s across one thousand managed servers. The BPPM 9.5 Service processes are generally stateless meaning the following. 1) The 9.5 Services do not cache namespace data and data points as in 9.0. The data is now streamed directly through to the BPPM Server. The server now gets every data point rather than only a snapshot every 5 minutes from the Service cached data points. 2) There are no adapters associated with PATROL data collection. a. All filtering of performance data is handled at the s. b. All filtering of events is handled at the s and if necessary in the event management cells. 3) The Service acts as a proxy to receive and forward both data and events that are sent to it from s. It also receives and Knowledge Module (KM) configuration data from CMA and passes that data to s. The following components can be optionally installed and configured on the Service host depending on whether or not they are needed in the environment. Before installing any of these additional components scalability and additional required resources must be considered. 1) Event Management Cell Event Management process installed locally on the same server with the Service. It is a recommendation and Best Practice to install the Event Management Cell on all Service hosts. 2) RT Server - This assumes the environment includes the PATROL Central Console which is not required. Refer to PATROL documentation for RT Server requirements. Note that the Console Server process should be installed on a separate machine. 3) Event Adapters - These work with the event management cell to consume non-patrol events. For example SNMP traps, etc. Significant non-patrol event collection should be dedicated to other event management cells as recommend in Best Practices for previous BPPM versions. The default event adapter classes, rules and files are installed with the cell that is installed with the Service installer. 4) and Knowledge Module (KM) for monitoring the Service host processes. Page 13 of 47

14 5) BMC Impact Web Services (IIWS). Certain processes that ran on the older Service hosts are no longer needed and should not be installed or used with a BPPM 9.5 Service node. These include the following. 1) A acting as a Notification Server 2) Service data collection Adapters (used in 9.0 and previous versions) 3) The BII4P3 or BII4P7 processes 4) The pproxy process The BPPM 9.5 Service is able to consume and forward both performance data and events. Technically the cell is not required in order to forward events to the BPPM Server. Therefore the cell technically does not have to be installed with the Service. For most environments BMC recommends propagating events from the Service to a lower tier event management cell. This is especially important in environments that meet any of the following conditions. 1) Involve more than a few thousand events in the system at any one time 2) Include multiple events sources other than PATROL 3) Has more than a few users 4) A medium or large environment involving more than 100 managed servers The event management cells allow you to further process events before sending them on to the BPPM Server. For example event enrichment, filtering, correlation, de-duplication, auto closure, etc. This type of event processing should be avoided on the BPPM Server as much as possible. Event processing in the BPPM Servers should be controlled and limited to the following. 1) Event Presentation of actionable events only 2) Collection of events for Probable Cause Analysis 3) Events used in Service Modeling Events sent to the BPPM Servers should be closely controlled and limited for the following reasons. 1) The event presentation in the BPPM Server should not be cluttered with un-actionable events that distract or otherwise reduce the efficiency of end users. 2) The new capability in BPPM 9.5 to view PATROL performance data in the BPPM Server without having to forward and store the data in the BPPM database will likely reduce the quantity of Page 14 of 47

15 parameters that are actually trended in the BPPM Server for most environments. This will likely increase the number of events propagated from PATROL for parameters that do not require baselines, but do require static thresholds. This increase will increase the load on the event management cell in the BPPM Server. 3) PATROL events are approximately twice the size in bytes compared to events generated in the BPPM Server. A larger volume of PATROL events will increase the memory consumption of the event management cell on the BPPM Server and will additionally increase BPPM Server start up time. Overall startup time for a BPPM Server at full capacity ranges from 15 to 20 minutes. 4) The automated event to monitor association introduced in 9.5 has slightly increased load on the event management cell that is embedded in the BPPM Server. The Central BPPM Server can act as a presentation server for all events processed in the child BPPM Servers. Events can be propagated from the Child BPPM Servers to the Central BPPM Server to accomplish this. Additionally, event management cells on the Service hosts should be integrated with the BPPM Server so that events in the remote cells are accessible in the BPPM Server web console under the Other Cells view. BMC recommends integrating the Child BPPM Servers with Remedy and other BMC products such as Atrium Orchestrator for event processing related to other products like these. These integrations should not be configured with the Central BPPM Server, except for the BMC Atrium SSO component. The following are additional Best Practices for Services, event management cells and the Service hosts. 1) Install an Service for each major network subnet. 2) Limit the usage of HTTPS between the Service nodes and the BPPM Server(s). HTTPS is not as scalable as HTTP and HTTPS requires more administration. 3) Do not send raw events directly to the BPPM Server. Every environment should have at least one lower tier event management cell. 4) Install the event management cell on all Service nodes. 5) Additional event management cells should not be installed on the BPPM Server. 6) Install additional event management cells on Service hosts and remote host as needed. 7) Do not configure IBRSD, Notifications, or other global event forwarding integrations on the lower tier event processing cells. Global event forwarding integrations should configured on the child BPPM Server(s). Page 15 of 47

16 8) The number and placement of event management cells should be based on the quantity of events, event source domains (secure zones, geography, etc), and major event sources. Always deploy multiple event management cells in the following situations. a. Large environments b. Geographically distributed managed infrastructure c. Large numbers of events d. When different event sources require different event management rules for example large numbers of SMNP traps compared to events from PATROL e. Significantly different event management operations are divided by teams 9) Configure display of remote event management cells in the BPPM server when necessary. 10) Install dedicated event processing cells to manage large volumes of events from common sources like SNMP traps, SCOM, and other significant sources of events. 11) Distribute event management cells as required, based on event loads and event sources. 12) Deploy event management cells close to or on the same node as the event sources for 3 rd party sources. 13) Filter, enrich, normalize, de-duplicate and correlate events at the lowest tier event management cells as much as possible before propagating to the next level in the event flow path. 14) Do not collect unnecessary events. Limit event messages sent from the data sources to messages that require action or analysis. 15) Do not try to use the event management cells as a high volume SNMP trap forwarding mechanism. 16) Use dedicated Service hosts for large domain data collection, for example vsphere, remote operating system monitoring and other large sources of data. 17) Install Service hosts close to the data sources that they process data for. Deploy by geography, department, business, or applications especially if multiple Services are required from a single source. 18) Do not collect excessive or unnecessary performance data. Review the need for lower polling intervals considering server performance and database size. 19) Do not collect trends for Availability metrics Page 16 of 47

17 Event & Data Flow Processing As already discussed above, both performance data and events are sent to the Service process from the s over the same communication path. The Service process then forwards events to the event management cell that is running locally on the same host with the Service. The event management cell further processes the events (filtering, enrichment, correlation, etc) and forwards the events to the BPPM Server. Performance data is sent from the Service process directly to an enterprise event correlation cell. The event correlation cell then forwards the events to the BPPM Server. events are now considered "Internal Intelligent" events in BPPM 9.5. Previous versions considered PATROL events as "External" events and they were managed like 3rd party events. PATROL Agent events in BPPM 9.5 are mapped to the instance object they belong to (previously the mapping was only at the device level). This monitor instance association of PATROL Events improves Probable Cause Analysis leveraging categorizations (e.g. Database, Application, Server, Network, etc.) The following are additional Best Practice recommendations for Services. 1) The Service installed with the BPPM Server (locally on the BPPM Server) should be configured as a Staging service. (In a POC environment it could instead be used for data collection.) Staging Services are discussed further in this document. 2) At least one remote Service node should be deployed for all environments. 3) BMC generally recommends installing the Service and event management cell in pairs so that each Service process has a corresponding event management cell installed on the same host. In this configuration events are propagated from the Service to the event management cell running on the same host. The install of an event management cell is an option available in the installer when installing the Service. 4) It is very important to maintain the event flow path so that all events from any one PATROL Agent are always processed through the same event management cell(s) (including cell HA pairs). This ensures event processing continuity where automated processing of one event is dependent on one or more other events from the same agent. A simple example of this type of processing is the automated closure of critical events that is triggered by Ok events for the same object that was in a state of critical alarm. If you do not maintain the same event flow path per agent through the same event management cell(s). Event correlation of all events from the same agent is not possible because the necessary events are not received and available in the same cell(s). 5) Some environments may require more than two IS nodes in a cluster and/or more than two Service nodes defined for each agent that is sending the data (events and performance) through a 3rd party load balancer to the Service nodes. This is acceptable as long as all events from any one agent always flow through the same HA cell pair so that event processing continuity as described above is maintained. For example if four Service nodes are clustered, then each node in the cluster should not have a cell Page 17 of 47

18 configured on it. Instead the cell should be on other systems (in an HA pair) so that the event path remains the same for all events coming from the agents that the cluster handles. Configuration regarding performance and event data that is sent from the s to the BPPM Server is defined in policies that are automatically applied to the desired s. assignment is defined in each policy configuration based specific criteria. The details of agent selection criteria per policy are discussed further in Central Management & Administration section of this document. PATROL events and performance data are completely controlled at the based on these policies. You have complete control meaning data, events, data and events, or no data and no events are controlled per parameter. These configuration settings can be edited and changed on the fly without having to rebuild any configurations or restart any processes. Page 18 of 47

19 Connection Details PATROL Data Collection BPPM 9.5 Architecture & Scalability Best Practices The diagram below illustrates the default ports on which connections are made regarding communications from the PATROL agents through the Service process to the BPPM Server for the BPPM 9.5 solution components. The direction of the arrows indicates the connection requests. Please review the product documentation for further details. BPPM Server Admin Cell Jserver Port 1827 CMA Agent Controller Event Cell Port Port 1828 Service Node Event Cell Port 1828 Service Port Port 3183 for (Self-Monitoring) Port 3181 Knowledge Modules Managed Node Port 3181 Knowledge Modules Data & Events Data Events Policies Legend Key Direction of arrows indicates connection requests. Note the following simplifications and changes from BPPM ) Port 3182 is no longer listening on the Service node for external connection requests. The BPPM Server communicates to port to send policies from CMA. 2) The number of processes and ports on the Service host has been reduced. There is no longer a pproxy process. Page 19 of 47

20 3) The Monitor the Monitor (MTM) KM does not discover and monitor the BPPM 9.5 Service and should not be used with the BPPM 9.5 Service. The built-in self monitoring is significantly enhanced with BPPM 9.5 and the MTM KM is no longer needed. However, the and Operating System KMs should be used for additional self monitoring and this is recommended. Administration & PATROL Consoles The diagram below illustrates the ports and connections related to the BPPM Administration and PATROL Consoles. BPPM Server Admin Cell Jserver Port 1827 CMA Agent Controller BEM Cell Port Port 1828 Service Node BEM Cell Port 1828 Service Port Port 3183 for (Self-Monitoring) RT Server Port 2059 PATROL Console Server PATROL Console Node PATROL Central Windows PCM PATROL Classic Console Managed Node Port 3181 Knowledge Modules Data & Events Data Events Policies Legend Key Authenication Direction of arrows indicates connection requests. An instance of the BPPM Administration console should always be installed on a separate machine from the BPPM Server. An instance of the BPPM Administration console is installed on the BPPM Server by default. This instance of the BPPM Administration console should only be used in an emergency if another instance is not available. Page 20 of 47

21 A PATROL Console is not required in every environment. The need to use PATROL Consoles and PATROL Configuration Manager (PCM) is reduced with BPPM 9.5 due to enhancements in BPPM Central Management and Administration. The PATROL Console should only be installed in environments where specific PATROL console functionality is required. The following are reasons to include PATROL Consoles and/or PCM. These will not apply to all environments. 1) The environment has a legacy PATROL implementation and the PATROL Console functionality needs to be continued for some period of time for migration and/or process related reasons. 2) Specific functionality in the PATROL Console is required that is not available in the BPPM 9.5 Console. Examples of functionality limited to the PATROL console include the following. a. Menu commands that generate reports b. Menu commands to initiate administrative actions that are built into the PATROL KMs and run against the managed technology c. Detailed analysis of certain events in PATROL If functions like these are not used and/or not required in IT management processes the PATROL Consoles may not be necessary in the production environment. Do not install a PATROL Console in the production environment if it is not needed. 3) Some PATROL Knowledge Modules (KMs) in use are not yet fully manageable in CMA. Check BMC s web site to verify which KMs are fully manageable in CMA as this list is constantly being updated. A list of compatible KMs can be found at the following URL. through+central+monitoring+administration 4) In certain situations detailed analysis of the and KM operations may be necessary for troubleshooting. This should be accomplished with a PATROL Central Console if it is required in production. The PATROL Classic Console should only be used for KM development and never used in a production environment. 5) Development of custom KMs to be loaded in CMA. This requires the PATROL Classic Console. It can also be used to analyze content in PATROL events at the PATROL agent and should be done in a development environment only. The PATROL Classic console should be used primarily for custom KM development. 6) When detailed understanding of a KM s functionality and how it is configured cannot be understood without analyzing the KM using the PATROL Console. This should be done in a development environment. Page 21 of 47

22 BPPM Administration Console The diagram below illustrates the connections between the BPPM Administration Console and other BPPM solution components. The ports listed are default ports and the arrows indicate the direction of the connection requests. BPPM Java Administration Console Port 1828 BEM Cell Port 1099 RMI Registry Port Jserver Port 3084 IAS Port 1828 BEM Cell Port 1827 Admin Cell Service Host BPPM Server Page 22 of 47

23 Central Management & Administration (CMA) BMC Recommends implementing one of two architectures for CMA. The two choices are listed below with their pros and cons. Choice One - Implement a single CMA Instance for all environments including Development, Test and. Pros Cons 1) The creation, testing, and deployment of monitoring policies into production are very easy because you do not have to copy or export/import any data. The application of policies to Development, Test and is simply managed in the policy s agent selection criteria. 2) It requires less infrastructure nodes and components. Only a single Staging Service host is needed. Only a single CMA instance is used. 1) This many not be supported in some sites where all the necessary connections between Development, Test and environments are not available or allowed to be connected over the network. 2) Due to the powerful ease of use, it is easier for administrators to make mistakes applying policies unintentionally to production. However, this can be managed. Choice Two - Implement a separate CMA Instance for the Development, Test and environments each. Pros Cons 1) Is supported in sites where all the necessary connections between Development, Test and environments are not available or allowed to be connected over the network. 2) Provides a platform and supports policy management methods that help prevent administrators from making mistakes when applying policies to production. 1) The creation, testing, and deployment of monitoring policies into production require more manual effort because you have to export/import policy data from Development to Test and from Test to. 2) Policies could get out of synch across the Development, Test, and environments if not managed properly. Keeping them up to date is more of a manual process supported by the export/import utility. Page 23 of 47

24 3) It requires more infrastructure nodes and components. The Development, Test, and environments should each have a dedicated Staging Service host and a dedicated CMA instance. IMPORTANT: Neither method supports seamless creation, testing, and production deployment of updates and deletion for existing policies. Updates and deletion of existing policies that are already in production should be created, tested, and populated to production leveraging the policy export/import capability. This topic is discussed in detail in the configuration best practices. In all scenarios, CMA communicates thorough the agent controller process on the BPPM Server(s) to the Service nodes. These implementation architecture options are not installation options. The CMA components are the same. These two implementation architecture options are simply choices in how you install CMA instances and connect them to the various BPPM Servers. Single CMA Architecture Overview The diagram below illustrates the high-level architecture for a single CMA instance in a multiple BPPM Server environment including Development, Test and. Central BPPM Server with CMA Policies Data & Events Legend Key Data Events Direction of arrows indicates connection requests. Staging Host QA BPPM Server Test BPPM Server BPPM Server N BPPM Server N+1 New Deployed QA Host Test Host Host N Host N+1 monitoring monitoring monitoring monitoring With the Single CMA Architecture a single Staging Service node is used in the agent deployment process for all agents. All BPPM Servers leverage the single CMA instance for all policy management Page 24 of 47

25 Multiple CMA Architecture The diagram below illustrates the high-level architecture for multiple CMA instances BPPM Server environments including Development, Test and. Development Central BPPM Server With CMA Test Central BPPM Server With CMA Central BPPM Server With CMA Manual Policy Export / Import Manual Policy Export / Import Development BPPM Child Server Development BPPM Child Server BPPM Child Server N BPPM Child Server N+1 Staging Service Development Service Staging Service Development Service Staging Service Service N Service N+1 New Deployed (Development) Development s New Deployed (Test) Test PATROL Agents New Deployed (Test) s s Legend Key Policies Data & Events Data Events Direction of arrows indicates connection requests. With this architecture each environment has its own dedicated CMA instance and Staging Service. All policy application between environments is supported by the policy export/import utility. Standalone BPPM Servers & CMA In most multiple BPPM Server environments the CMA module will be installed with a Central BPPM Server. However, it is possible to install CMA with a stand-alone BPPM Server and then manually register the additional BPPM Servers with CMA after the install. The following are reasons for installing CMA on stand-alone BPPM Servers and not leveraging the Central Server capability. 1) The top tier BPPM Server is only needed to provide an enterprise event console. Page 25 of 47

26 2) BPPM Central Server functions are not needed. a. Single point of entry for service model visualization. b. Enterprise level map view. CMA Architecture Details The diagram below illustrates the default ports and connectivity that support Central Management and Administration across multiple BPPM Servers. The arrows indicate the direction from which the connections are established. BPPM Server with CMA JMS Port 8093 Web Services Port 80 / 443 BPPM Server (Child or Leaf) JMS Port 8093 BPPM Server (Child or Leaf) JMS Port 8093 Agent Controller Port Web Services Port 80 / 443 Web Services Port 80 / 443 Agent Controller Port Service Host Service Host Service Port Port 3183 Service Port 3183 Port Managed Host Port 3181 Managed Host Port 3181 Knowledge Modules Knowledge Modules The detailed architecture above applies to both the Single CMA Architecture and the Multiple CMA Architecture. Page 26 of 47

27 Staging Services Overview & Functionality The Staging Service has been introduced in BPPM 9.5. The Staging Service provides a single point in the environment where all newly deployed s can register into the BPPM solution stack. The staging process can leverage the Service on the BPPM Server. A Staging Service supports a smoother process for deploying s into BPPM environments in three major ways. 1) It eliminates the need to manage the decision and assignment of s to production Service node separate from the deployment process. (This assumes an environment that includes multiple production Service instances.) When you leverage a Staging Service this decision and the assignment is automated as part of the deployment process. 2) It supports a smoother process for managing policies across Development, QA, Test, and environments. 3) It reduces the number of PATROL silent install packages that have to be created and maintained. silent install packages are created so that the Staging Service is defined in the package. No other Service is defined in the install packages. Although technically possible, it is recommended as a best practice that other Service instances not be defined in the install packages. When a package is deployed and installed, the agent will check in through the Staging Service. When the agent checks in, the Central Management & Administration module evaluates agent selection criteria in a Staging Policy and uses that data to automatically assign a data collection Service (or Service cluster) to the agent. The agent selection criterion can include any one or any combination of the following. 1) A tag defined in the agent configuration 2) Hostname that the agent is running on 3) Operating System that the agent is running on 4) IP Address or IP Address range that the agent is running on 5) Agent Port 6) Agent version 7) Service that the agent is already assigned to (assuming it is already assigned) 8) BPPM Server that the agent is already assigned to (in this case it is through a Staging Service) Page 27 of 47

28 Service policies are the only CMA policies that are applied through a Staging Service. Monitoring policies are not applied though a Staging Service. Additionally, Staging Service policies in CMA are only applied through a Staging Service instance. The architecture of network connections (communication protocol, ports, etc) between the Staging Service, s, and the BPPM Server are technically the same as with other Service instances. The following are best practices for Staging Service nodes. 1) Do not attempt to configure agents so that performance data and/or events are sent to a Staging Service. 2) Staging Services must not be mixed with Data Collection Services. They must be configured, used, and managed separately from Data Collection Services. 3) Configure the Service on the BPPM Server as a Staging Service. Do not use it for data collection. 4) If firewall rules and security prevent using the Service on the BPPM Server as a Staging Service, deploy a Staging Service into the managed zone or zones. 5) Setup a single Staging Service for each environment, for example one for Development, one for Test, and one for. Or, if you have a single CMA instance for all environments, setup a single Staging Service for the entire implementation when possible. 6) Consider high availability for Staging Services. Refer to the High Availability section for more information. Page 28 of 47

29 Staging Process Illustration The diagrams in Steps 1 through 3 below illustrate the process of utilizing a Staging Service. Initial Agent Deployment BPPM Server Policies Data & Events Data Events Legend Key Direction of arrows indicates connection requests. Staging Service General Service Domain Service New Deployed general monitoring Dedicated PATROL Agent & IS nodes for domain monitoring The diagram above illustrates there different Service nodes and how they are used as follows. 1) The Staging Service is used strictly for introducing new agents to the BPPM Server. (An Service has to be configured to work as a Staging Service.) 2) The General Service is used for collecting data from various deployments of PATROL agents that are installed locally on the managed nodes. The term General is a description of how the Service is used and is not a configuration. 3) The Domain Service is used for collecting data from s that provide large volumes of data from a single source. Examples are VMware vcenter, PATROL Remote Operating System Monitoring, NetApp, etc. The term Domain is a description of how the Service is used and is not a configuration. The agent introduction process works as follows. A newly deployed silent install package is installed as shown above. The install package for the contains configuration data telling the how to connect to the Staging Service. When the new agent starts for the first time, it registers with CMA through the Staging Service. CMA then applies a Page 29 of 47

30 Staging policy to the agent based on agent selection criteria in the policy. Agent selection criteria define what agents the policy should be applied to. The Staging policy only contains agent selection criteria and information that defines connectivity information for a data collection Service node (or Service cluster). No other agent and/or KM configuration data can be defined in a Staging policy. Service Policy Application BPPM Server Policies Data & Events Data Events Legend Key Direction of arrows indicates connection requests. Staging Service General Service Domain Service New Deployed local monitoring Dedicated PATROL Agent & IS nodes for domain monitoring After receiving the Staging policy, the newly deployed agent switches to the data collection Service node (or Service cluster) defined in the Staging policy. (The switch is represented by the blue arrow in the diagram above.) The agent then receives any monitoring polices defined in CMA that match each Monitoring policy s agent selection criteria. Page 30 of 47

31 Monitoring BPPM 9.5 Architecture & Scalability Best Practices BPPM Server Policies Data & Events Data Events Legend Key Direction of arrows indicates connection requests. Staging Service General Service Domain Service local monitoring Dedicated PATROL Agent & IS nodes for domain monitoring The agent starts monitoring and continues to receive any updates to existing monitoring policies and new monitoring policies that match the monitoring policy s agent selection criteria. NOTE: Agents do not move from Development to Test, and then to. All agents should first check in with the appropriate Staging Service, then move to their data collection Service. This supports the concept of creating install packages for Development and Test only, separate from. This topic is discussed further in the BPPM 9.5 Configuration Best Practices. Page 31 of 47

32 Staging & Policy Management for Development, Test and Single CMA Instance Deployments A single Staging Service and a single CMA instance can be used to support multiple BPPM Servers. The diagram below illustrates how this is architected for Development, Test, and BPPM Server environments. Central BPPM Server with CMA Policies Data & Events Legend Key Data Events Direction of arrows indicates connection requests. Staging Host Development BPPM Server Test BPPM Server BPPM Server N BPPM Server N+1 New Deployed Development Host Test Host Host N Host N+1 monitoring monitoring monitoring monitoring All policies include agent selection criteria that allow you to completely control what policies are applied to any and all s across the entire environment spanning Development, Test and. This allows you to install a single CMA instance for the entire environment, and it eliminates the need to recreate policies in production after they have been created in development and tested, etc. One or more of the following agent assignment configurations in the policies is defined and edited to accomplish this. 1) BPPM Server that the agent is assigned to (Best Practice) 2) Service that the agent is assigned to 3) A tag defined in the agent configuration Page 32 of 47

33 4) Hostname that the agent is running on 5) IP Address that the agent is running on The easiest method is to include the appropriate BPPM Servers in the agent selection criteria for the policy as shown in the screen below, at the proper time. Simply add the Test and BPPM Servers to the policy agent selection criteria when you are ready to apply the policy to those environments. This simplifies the process of moving configuration from QA to Test, and finally to. It also ensures a policy is not applied to any agents in production until it has been tested and validated. The following outlines the process as an example referencing the screen shot above. Phase 1 Only the BPPM Server named BPPMRHEL62-HM-QA is included in the agent selection criteria when the policy is first created. Phase 2 The BPPM Server named BPPMRHEL62-HM-TEST is added to the agent selection criteria with an OR after the policy has been validated in QA. Page 33 of 47

34 Phase 3 The BPPM Server named BPPMRHEL62-HM-PROD is added to the agent selection criteria with an OR after the policy has been tested and validated in the BPPMRHEL62-HM-TEST environment and is ready to be applied to production. WARNING: At least one BPPM Server must be included in the agent selection criteria in order to control which BPPM Server environment(s) the policy is applied to. If you do not include at least one BPPM Server the policy will be applied to all agents, across all BPPM Servers that match the agent selection criteria of the policy. Additionally the multiple BPPM Server values must be grouped () and related with a Boolean OR as shown above. If you use a Boolean AND to relate the agent selection criterion the policy will not be applied because an agent cannot register with multiple BPPM Servers. WARNING: leveraging the BPPM Servers in agent selection criteria is powerful and has far reaching, global implications. If you mistakenly add a production BPPM Server to the agent selection criteria for a policy the policy could be unintentionally applied to 100s or 1000s of agents in production. Therefore it is extremely important that this process be managed carefully. IMPORTANT: Updates and deletion of existing policies will apply to all agents that the policy s agent selection criteria match. Consequently it is not possible to test edits to policies that currently apply to production without impacting production using the process outlined above. Separate policies should be created to test edits in the development and test environments leveraging the export/import utility. The details of this topic are discussed in the configuration best practices. Leveraging tags in policies can also be used to further control what agents policies are applied to. Tags should be used to provide a second level of protection to prevent policies still in development or test from being applied to production accidentally. Leveraging tags this way forces the user to not only add the appropriate BPPM Server to the policy selection criteria, but they also have to add the appropriate tag. This helps prevent the user from accidently picking the production BPPM Server and saving the policy when they did not mean to. Additionally, tags can be used to provide greater granularity for policy assignment where the other agent selection criteria is not enough. BMC recommends leveraging precedence in policies so that production policies have the highest precedence and will not be superseded by development or test policies. If you follow this recommendation you will also have to adjust the policy precedence when you want to move it from development to test, and finally from test to production. The configuration topics above are discussed further in the BPPM 9.5 Configuration Best Practices. Page 34 of 47