E n d To E n d M o n i t o r i n g H y p e r i c H Q I n t e g r a t i o n W i t h O p e n N M S

E n d To E n d M o n i t o r i n g H y p e r i c H Q I n t e g r a t i o n W i t h O p e n N M S David Hustace, Jeff Gehlbach The OpenNMS Group, Inc. Revision History Version Date Author Comments 1.0 1-Jan-2008 David Hustace Initial version 1.1 29-Apr-2009 Jeff Gehlbach Updated for OpenNMS 1.6 and HQ 4.1 Page: 1 of 13

Executive Summary There are two Java based open source network management software projects available today from SourceForge: Hyperic HQ and OpenNMS. Both applications are truly developed for scalability and integration in the enterprise and both are written in Java and are professionally developed and commercially supported. Even though at first glance it would appear each of these projects serves the same purpose, as it turns out, these projects are actually more complementary than competitive with only a fractional amount of overlap. Hyperic HQ is used by many organizations to support the monitoring of applications and their platforms leveraging its strong agent based technology. OpenNMS is used by organizations monitoring network infrastructure. The overlap is where Hyperic s agents can be used to test local network infrastructure and OpenNMS can be used to communicate with agents and monitor platforms. OpenNMS is designed to be is strongly agent-less and Hyperic HQ s architecture is strongly agent-based. Often the argument for or against agent vs. agent-less monitoring is supposed by someone, somewhere, on the Net. I suppose a hybrid End-to-End solution using both agent-less and agent-based monitoring systems can best serve the IT operations staff challenged with monitoring today s complex and diverse web-based applications. To support this supposition, the OpenNMS project has written plugins for Hyperic-HQ that will natively interact with OpenNMS. This allows OpenNMS distributable synthetic transaction monitors coupled with Hyperic s expert application and platform monitoring technologies to provide a comprehensive view from the vantage point of each application user. Hopefully, this integration will be only the first step in a an ongoing collaborative integration project. The OpenNMS project is constantly releasing integration software for other management and trouble ticketing systems. We hope this encourages and fosters more collaboration in this space. As always, since the OpenNMS project is an engineer s project, we speak with our feet. What follows are the technical details for integrating Hyperic HQ with OpenNMS. Enjoy! Page: 2 of 13

Concept Integration with another monitoring application begins with sharing alert information. However, in order to make sense of (correlate) these shared alerts from other monitoring applications, inventory must be also be shared or related (for a complete design, see NGOSS 1 ). The OpenNMS configurations and Hyperic HQ (HQ) software plugins included in this integration, accomplish these requirements from the perspective of OpenNMS. OpenNMS provides an API for synchronizing monitored elements from one or more other monitoring systems, such as HQ, and provisioning databases. It also provides flexible means for correctly associating alerts from external systems with the correct entities imported into OpenNMS via an Event Bus adaptor called the Event Translator. The HQ software is used to forward notifications from HQ to OpenNMS as an HQ alert action and a servlet is provided for HQ (using HQ s cool HQU Groovy plugin framework) to export the platform inventory as an OpenNMS Model Import. Alerts are sent to OpenNMS via the OpenNMS TCP EventProxy and are then associated with an OpenNMS node entity that was imported from HQ using the translator. Platforms imported as OpenNMS nodes retain their identity (platform ID) as an attribute on the OpenNMS node called foreign ID. When an alert is received from HQ, the HQ alert event is persisted for historical purposes and then translated into an event that is assigned to the OpenNMS node by matching the alert event s platform ID parameter with the node s foreign ID. Confused yet? Don t worry, it s all been configured for you. 1 http://www.tmforum.org/solutionframeworks/1911/home.html Page: 3 of 13

Preparing OpenNMS for Integration There are two main components of this integration: On the OpenNMS side: The Hyperic Agent service definition and monitor Hyperic HQ service definition and monitor Definition of the new Hyperic Events and Alarms Hyperic Alert Event Translator configuration Eventd TCP listener On the Hyperic HQ side: Hyperic HQU platform-export plugin (already done for you) Hyperic Groovy based OpenNMS alert action definition Configuration of alerts to send events to OpenNMS Version Note: The following instructions pertain to OpenNMS release from 1.6.0 onward. The Completed OpenNMS Configuration The default OpenNMS install comes with all but one security-related setting configured for the integration. The following is an itemized that defines all the configurations included in the default installation based on the OpenNMS hyperic-integration examples directory: 1. Verify HypericAgent service is defined in capsd-configuration.xml (already done for you; see details) 2. Verify HypericAgent service is defined in a polling package in poller-configuration.xml (already done for you; see details) 3. Verify the HQ event translation is configured in translator-configuration.xml (already done for you; see details) 4. Verify the HQ event definition is defined in $OPENNMS_HOME/etc/events/ Hyperic.events.xml (already done for you; see details) 5. Verify that there is a reference to it and that a definition for the translated event are found in $OPENNMS_HOME/etc/eventconf.xml 6. Configure the OpenNMS Eventd TCP listener to listen for connections on all interfaces (must be done once manually; see details) More Cool OpenNMS Preparation Stuff For security reasons, the TCP listener used by OpenNMS event daemon to receive XMLformatted events binds by default only to the software loopback interface at IP address 127.0.0.1. If HQ is on a separate server from OpenNMS, you will need to configure this listener to bind to all IP interfaces. Edit the file $OPENNMS_HOME/etc/eventd-configuration.xml and change the TCPAddress attribute of the <EventdConfiguration> element from 127.0.0.1 to 0.0.0.0. You must restart OpenNMS for this change to take effect. Page: 4 of 13

Possibly Required Changes to HQ Service Monitoring in OpenNMS A service monitor has been added to the OpenNMS default installation that supports the monitoring of the HQ server itself and is called HypericHQ. This is a synthetic transaction monitor that connects to the HQ server and verifies a successful login and logout. Other pages may be added for further testing and verification of the server. Response time is recorded so that thresholding can be used to alert for possible monitoring outages of the HQ application and server itself. The default hqadmin login and password are used, so change these to match the configuration of your HQ server. You may also have to change the scheme to https if your HQ server is configured not to offer its web interface via unsecured HTTP. Security Note: Scanning for the HypericHQ service has been disabled to avoid sending your HQ admin login credentials to any host configured to listen on the default HQ port (7080). It is suggested that you create a provisioning group for your HQ servers to be monitored since scanning for this service is disabled by default. Define the server with the HypericHQ service and import. A sample definition can be found in the OpenNMS hyperic-integration contrib directory called imports-opennms-admin.xml. Manually Provisioning HQ Servers in the OpenNMS Provisioning Groups interface Best Practice Note: Monitoring of entities that support your network management environment (SMS gateways, e-mail, ticketing systems, etc.) is considered a best practice Feature Availability Note: The Event Translator, Alarms, Model Importer, and Provisioning Groups features are available in all stable OpenNMS releases starting with 1.6.0. Page: 5 of 13

Preparing Hyperic HQ for Integration Integration from the HQ side requires that the OpenNMS alert mechanism (included with HQ but disabled by default) be enabled. Version Note: These instructions pertain to Hyperic HQ version 4.1.1. Enabling the HQ OpenNMS Alert Definition HQ includes an alert mechanism for sending events to an OpenNMS server, but the default HQ configuration does not include a definition enabling this alert mechanism to be used. To enable this alert mechanism, edit the following file on your HQ server: $HQ_SERVER_HOME/hq-engine/server/default/deploy/hq.ear/alertdef-prop-service.xml Near the bottom of this file is an XML <attribute name= Properties > tag that contains an empty definition for the value of a property called actions. Simply add the name of the OpenNMS event alert action class as the value of this property: actions = \ org.hyperic.hq.bizapp.server.action.integrate.opennmsaction The HQ server must be restarted in order for the new alert mechanism to be available for use. Configuring Alerts To configure HQ to sent alerts for certain conditions as OpenNMS events, simply configure an alert against some resource as you would normally do to notify an HQ user or other recipients. Once the OpenNMS alert action is enabled, you will see a new tab in the alert workflow labeled OpenNMS. Fill in the IP address of your OpenNMS server in the Server: text field. If you have configured OpenNMS Eventd TCP listener to bind to a port other than the default of 5817, enter that port number in the Port (default 5817): text field. Page: 6 of 13

OpenNMS Alert Definition in HQ Feature Availability Note: The open source community edition of Hyperic HQ requires that resource alerts be configured individually. Automatic application of alert definitions to groups of resources or globally across resources of a given type is available in HQ Enterprise Edition. The Integration in Action Importing HQ Platforms to OpenNMS Nodes A servlet ships with Hyperic HQ that makes HQ s inventory of platforms available in an XML format that OpenNMS can use. The OpenNMS Model Importer, once configured to use the URL of this exporter servlet, will synchronize HQ s platforms as OpenNMS nodes. You can add the URL of this servlet in the Model Importer service s properties file ($OPENNMS_HOME/etc/modelimporter.properties) for scheduled imports on a cron-like schedule, or you can use a tool such as wget or curl to fetch the servlet s output for manual use with the OpenNMS Provisioning Groups web interface. There is a sample script (imports-hq.sh) provided in the OpenNMS contrib/hypericintegration directory that shows this latter usage. Import Using Provisioning Groups Interface Edit the import-hq.sh located in the $OPENNMS_HOME/contrib/hyperic-integration directory and run it to create the import-hq.xml file in $OPENNMS_HOME/etc. Then, using the Provisioning Groups Interface, import the nodes to OpenNMS. Page: 7 of 13

The Group Name will be, and must be, HQ to match the foreign-source attribute inside the import file from the servlet and the name of the file itself, imports-hq.xml. Configure model-importer.properties Adjust the cron schedule for imports to your liking and add the proper URL: importer.importurl=http://myhqserver.mydomain.com:7080/hqu/opennms/exporter/index.hqu importer.importschedule=1 0 0 * *? Making these changes will schedule an import from HQ into opennms daily at one second after midnight. Screen Shots After OpenNMS 1.6.x is installed and the OpenNMS event alert action is enabled in HQ, OpenNMS will begin processing alerts from HQ and creating alarms. Just as with all OpenNMS alarms, the Hyperic HQ alarms will be integrated into the automation system and notification systems and can be used to create help desk tickets in an external system via OpenNMS trouble ticket integration API. The following is a screen shot of the event received from HQ and the translated event being associated with the node that was imported from HQ. The OpenNMS nodes are synchronized with HQ platforms during each import. The HQ alert event, and translated event matching the node imported from corresponding HQ platform, in the OpenNMS event browser Page: 8 of 13

The next screen shot is from the OpenNMS Alarms interface showing that the translated event has spawned an alarm in OpenNMS. Note that the alarm has been raised 110 times (the event reduction feature of OpenNMS... we hope that your staff fixes this problem before it gets this bad!) The HQ alert, now represented as an alarm (note the Count column) in the OpenNMS alarm browser Notice that the Log Message is formatted as a hyperlink. Clicking this link will take you directly to the alert in HQ console. Each time the event is reduced to the single alarm, the count will increase and the URL will update to reference the most recent occurrence of the alert in HQ. The HQ alert viewed in HQ by clicking the drill-back link from the OpenNMS alarm details Viewing the details of the alarm will present standard OpenNMS workflow: acknowledgments, escalations, clears, and trouble ticket system integration (not shown here). Page: 9 of 13

Details for an alarm derived from an HQ alert, viewed in the OpenNMS alarm browser Notice in the Description text another hyperlink whose text is just the name of the node to which the alarm pertains. This hyperlink will direct the user to the page for the corresponding platform in the HQ console. Page: 10 of 13

Default OpenNMS Configuration Examples Define the services in capsd-configuration.xml These definitions are already present in the default configuration files. <protocol-plugin protocol="hypericagent" class-name="org.opennms.netmgt.capsd.plugins.tcpplugin" scan="on"> <property key="port" value="2144" /> <property key="timeout" value="2000" /> <property key="retry" value="1" /> </protocol-plugin> <protocol-plugin protocol="hyperichq" class-name="org.opennms.netmgt.capsd.plugins.httpplugin" scan="off"> <property key="port" value="7080" /> <property key="timeout" value="2000" /> <property key="retry" value="1" /> </protocol-plugin> Define the monitoring of Hyperic services in poller-configuration.xml These definitions are already present in the default configuration files. <service name="hypericagent" interval="300000" user-defined="false" status="on"> <parameter key="retry" value="1" /> <parameter key="timeout" value="2200" /> <parameter key="port" value="2144" /> </service> <service name="hyperichq" interval="300000" user-defined="false" status="on"> <parameter key="retry" value="1" /> <parameter key="timeout" value="3000" /> <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response" /> <parameter key="rrd-base-name" value="hyperic-hq" /> <parameter key="ds-name" value="hyperic-hq" /> <parameter key="page-sequence"> <page-sequence> <page path="/login.do" port="7080" successmatch="(hq Login) (Sign in to Hyperic HQ)" /> <page path="/j_security_check.do" port="7080" method="post" failurematch="(?s)(the username or password provided does not match our records) (You are not signed in)" failuremessage="hq Login Failed" successmatch="hq Dashboard"> <parameter key="j_username" value="hqadmin" /> <parameter key="j_password" value="hqadmin" /> </page> <page path="/logout.do" port="7080" successmatch="hq Login" /> </page-sequence> </parameter> </service> <service name="mysql" interval="300000" user-defined="false" status="on"> Define the event translations for HQ alerts in translator-configuration.xml These definitions are already present in the default configuration files. <event-translation-spec uei="uei.opennms.org/external/hyperic/alert"> <mappings> <mapping> <assignment name="uei" type="field"> <value type="constant" result="uei.opennms.org/internal/translator/hypericalert" /> </assignment> Page: 11 of 13

<assignment name="nodeid" type="field"> <value type="sql" result="select n.nodeid FROM node n WHERE n.foreignid =? AND n.foreignsource =?"> <value type="parameter" name="platform.id" matches=".*" result="${0}" /> <value type="constant" result="hq" /> </value> </assignment> </mapping> </mappings> </event-translation-spec> Define the HQ alert event in Hyperic.events.xml These definitions are already present in the default configuration files. <events> <!-- Start of Hyperic Events --> <event> <uei>uei.opennms.org/external/hyperic/alert</uei> <event-label>opennms defined event: An event has been received from Hyperic HQ</event-label> <descr> This event is generated by an integration developed for sending events to OpenNMS via the Hyperic Alert Notification mechanism.<p> <p>long reason: %parm[action.longreason]% </p> <p><a href="%parm[resource.url]%" > %parm[resource.name]% </a></p> </descr> <logmsg dest='logndisplay'> <p>hyperic HQ Alert: %parm[action.longreason]% </p> </logmsg> <severity>warning</severity> <alarm-data reduction-key="%uei%:%dpname%:%parm[platform.id]%:%parm[resource.id]%" alarm-type="1" autoclean="false" /> </event> </events> Define the HQ translated event in eventconf.xml These definitions are already present in the default configuration files. <event> <uei>uei.opennms.org/internal/translator/hypericalert</uei> <event-label>opennms defined event: An event received from Hyperic has been translated</event-label> <descr> This is a translated Hyperic Alert to associate with OpenNMS entity..<p> <p>alert reason: %parm[action.shortreason]% </p> <p>alert reason: %parm[action.longreason]% </p> <p><a href="%parm[resource.url]%" > %parm[resource.name]% </a></p> </descr> <logmsg dest='logndisplay'> <p><a href="%parm[alert.url]%" > Hyperic Alert: %parm[action.longreason]% </ a></p> </logmsg> <severity>minor</severity> <alarm-data reduction-key="%uei%:%dpname%:%nodeid%:%interface%:%service%:%parm[resource.name]%: %parm[alertdef.name]%" alarm-type="1" auto-clean="false" /> </event> Page: 12 of 13

Configure the Eventd TCP listener This step represents a change to the default configuration files, and must be completed for the integration to work. Only the value of the TCPAddress attribute (highlighted distinctively below) needs to be changed. <EventdConfiguration TCPAddress="0.0.0.0" TCPPort="5817" UDPAddress="127.0.0.1" UDPPort="5817" receivers="5" getnexteventid="select nextval('eventsnxtid')" getnextalarmid="select nextval('alarmsnxtid')" socketsotimeoutrequired="yes" socketsotimeoutperiod="3000"> </EventdConfiguration> Page: 13 of 13