Storage Health and Event Monitoring

HP OpenView Operations, Service Desk, Insight Manager and Storage Essentials Integration Storage Health and Event Monitoring HP Storage Essentials, System Insight Manager, Openview Operations and Service Desk Integration... 2 The goal... 2 Architecture... 4 Configuration Steps... 6 Step 1: Setup SNMP trap forwarding destinations for storage switches and device managers... 6 Step 2: Install the Openview Operations agent on the SIM and/or SE management server(s)... 8 Step 3: Configure System Insight Manager and Storage Essentials SPI for Openview Operations... 9 Default event forwarding... 9 Custom event forwarding... 10 Step 4: Importing Configuration Items into Service Desk... 10 Event Monitoring Details... 11 System Insight Manager... 11 Storage Essentials... 11 Openview Operations... 14 Openview Service Desk... 15 Customer case studies... 15 Customer A Genentech... 15 Customer B DHL... 18 System Requirements... 18 For more information... 18

HP Storage Essentials, System Insight Manager, Openview Operations and Service Desk Integration The goal Main goals: Providing linkages for information flow between Storage Essentials and other HP OpenView products: events, elements, actions, maps, tools, configuration, etc. Discover SIM and SE services on management server and agents Forward storage events from HP System Insight Manager to HP Openview Operations Business benefits: Monitor availability of Insight Management Agents and HP SIM services Monitor availability of SE Cim Extensions and SE Appstor service Associate hardware event status with applications and service availability Auto-discover ProLiant Servers running Insight Management Agents for Windows HP SIM policies auto-deployed with OVOW for event forwarding, acknowledging and clearing HP SIM and SE policies include automated event filtering for redundant sub-systems Links to HP Systems Insight Manager, Lights-Out management, System Management Homepage, Storage Essentials via HP-SIM Example: Enterprise Virtual Array events are logged under the controller event log. The Command View GUI will display critical errors at different levels. These will need to be followed up by a local storage admin, or forwarded onto the Openview Operations Console to be followed up at Helpdesk level. Note that EVA ISEE Phone Home support is available to forward these critical events to HP Services Support. When a hardware issue occurs, we would see the errors logged on Command View EVA GUI, and in the EVA controller event log:

Figure 1 EVA Command View Event Monitoring These errors will come into the System Insight Manager Event Viewer, and Storage Essentials Event Manager as in Figure 3, and will be forwarded by the Openview Operations SPIs to be shown in the Openview Console as in Figure 4. Figure 2 System Insight Manager Events

Figure 3 Storage Essentials Event Manager Figure 4 Openview Operations Event Panel Architecture The OVO agent: OVO intelligent agents are configured on the HP Systems Insight Manager, HP Storage Essentials CMS, and managed servers, managed nodes in the OVO management domain. Monitors the availability of the Storage Essentials processes at periodic intervals. Forwards appropriate events to the administrator if the processes are not running.

The System Insight Manager and Storage Essentials SPI: Troubleshoot problems in the storage network before they become serious enough to affect applications and end-users. Storage events are stored in the HP Systems Insight Manager s database. Discover Storage Essentials related services. This enables administrators to perform root-cause analysis of the inherent Storage Essentials services. Monitors the Storage Essentials central management application, the AppStorManager process on Windows node. Monitors the Storage Essentials CIM Extension process, on UNIX managed nodes. Offers tools to start, stop, and view status of Storage Essentials AppStorManager service. Offers tools to start, stop, and view status of Storage Essentials CIM Extension process on UNIX managed nodes. Offers tools to view status of Storage Essentials CIM Extension process on Windows managed nodes. Discovers HP Storage Essentials services. SE Oracle SIM SQL OVOW SQL OVO agent OVO agent SIM SPI SE SPI HP-SIM/SE Connector HP-SIM Integration online Unix host IM agent CIME agent OVO agent Near-online Windows host IM agent CIME agent OVO agent offline tape Figure IT Operations Storage Management Service Desk integration: Mechanism to import both server and storage elements from SE and HP SIM into Service Desk Elements in SD are known as Configuration Items (CI) Forward storage events from both SE and HP SIM to SD Events in SD are known as Incidents

Business benefits: Server and Storage Elements are imported into the customer s central CMDB and can be tracked to: service level agreements service and infrastructure dependencies Service state and user relationships Storage events (incidents) are associated to the corresponding CI View impacted Services, Service Levels, and Users Initiate Service Calls Service Desk Incid CMD CI Events, Acknowledgement OVO Management Server Event Tools Servi Events Configuration Items I Events, Acknowledgement Events can be sent two ways: Direct to SD HP SIM/SE Management Server Indirect via Operations SAN Hosts, HBAs, FC Switches, Storage Figure IT Service Desk Integration Configuration Steps Step 1: Setup SNMP trap forwarding destinations for storage switches and device managers Setup the SNMP trap destination for switches to be the SIM/SE management server(s).

Setup EVA Command View or any other Device Management tools to forward SNMP traps to our System Insight Manager management server. Any WEBES based SMIS indications will be received natively into Storage Essentials. You will need to do this for each array being managed by CV. The events will come in via SIM under the all events folder and may only be associated with the CV EVA server and not the individual arrays. Configure the Command View Server with the trap destination and select all events to be forwarded as in Figure 2. Figure 2 EVA Command View SNMP trap destination Decide what events will be forwarded to the SIM-SE server:

Figure 3 EVA Command View SNMP event notifications To receive events from the Device Manager, the public community string needs to be the same on the Device Manager side and on the SE side. Note that this is case sensitive. See also the Hp engineering advisory on how to setup events in CVEVA and SE. In the Device Manager, for instance EVA Command View, change the SNMP community string in nsaserver.ini: ; Element Manager SNMP Trap head Details. [SNMP_TRAP_HEAD] ; Head Part of each SNMP Trap. Authority = public In Storage Essentials jboss.properties configuration file, make sure the trap listener community string is the same as your Device Manager s. Note that this is case sensitive: # CIMOM starts an SNMP trap listener for any provider to use to receive traps. # By default, the trap listener uses a community string of "public". Use this # property to change the default community string for incoming SNMP traps. cimom.snmptraplistenercommunitystring=public Step 2: Install the Openview Operations agent on the SIM and/or SE management server(s) The Openview agent is installed on the System Insight Manager and Storage Essentials server or servers. This can be a dual deployed installation with SIM on SQL or Oracle on a remote server, or SIM and SE on Oracle installed on the same server. The Insight Manager and Storage Essentials management servers will be added as managed nodes to the Openview Operations server.

The agent will have the Unique ID of the management server in the OPCINFO file: C:\Program Files\HP OpenView\Installed Packages\{790c06b4-844e-11d2-972b-080009ef8c2a}\bin\OpC\install ############################################################################## # File: opcinfo # Description: Installation Information of ITO Managed Node # Package: HP OpenView IT/Operations # Status: ############################################################################## OPC_INSTALLED_VERSION A.07.31 SVCDISC_INSTALLED_VERSION A.07.28 PERF_INSTALLED_VERSION A.07.23 COMM_INSTALLED_VERSION 2.6.8.0 OPC_MGMT_SERVER server.domain.com OPC_INSTALLATION_TIME 09/01/2006 14:12:18 OPC_SG FALSE OPC_NO_CFG_RQST_AT_STARTUP TRUE OPC_EXEC_USER SYSTEM Step 3: Configure System Insight Manager and Storage Essentials SPI for Openview Operations Each of the hosts that are managed as Storage Essentials hosts should also have the Openview Operations agent installed. The SPI will monitor the SE host agent services status, and stop and restart it. The host with Insight Manager and Storage Essentials agents will be added as managed hosts to the Openview Operations server. The System Insight Manager and Storage Essentials SPIs will be installed and configured from the Openview Operations Windows or Unix servers. A default event forwarding configuration or custom event forwarding filters can be setup. It is important to understand what events will be forwarded, as after configuration of either custom or default event forwarding, reconfiguration will have to be done if proven to receive too many events. A few considerations: If default event forwarding is chosen to be setup, this may result in message storms when devices are set to forward a large amount of events, for instance when the switch level is set to forward all critical and warning alerts to a high configured amount, this may result in thousands of SNMP alerts received in System Insight Manager. If default event forwarding is chosen and custom filters are added to be forwarded, this may result in double events to be received in Openview Operations. If custom event forwarding is chosen, a separate event filter is setup per device or group of devices, for the event type, severity, subsystem, or other custom selections. Additional filters can be added at any time. When configuring per single device, a lot of filters may have to be configured. Default event forwarding Events are forwarded by running the following configuration tools: Setup default event forwarding policies using the tool Forward Important Events.

Configure bi-directional event clearing: When using default event query and forwarding, run the tool Forward Cleared Important Events Custom event forwarding Events are forwarded by running the following configuration tools: Setup custom event forwarding policies using Custom Event selections in SIM and using the tool Create Events Task Configure bi-directional event clearing: When using custom event query and forwarding, run the tool Create Events Task with the selection made in SIM events panel to be cleared Figure System Insight Managercustom event selection Step 4: Importing Configuration Items into Service Desk

Event Monitoring Details System Insight Manager Events can be customized to forward, acknowledge and all critical and major events to Openview Operations, or use custom filters. Note that if all events are forwarded, message storms can t be avoided in System Insight Manager. Figure 5 System Insight Manager Event Management Storage Essentials HP Storage Essentials will display critical errors that are logged for the devices on the System Topology Map.

Figure EVA Command View Event Monitoring

Figure Storage Essentials Event Management Figure Storage Essentials Event Details

Openview Operations Note that only the hosts will appear on the service maps. The events for all storage devices will appear on the Event Panel and can be assigned to a manually created storage device. Figure Openview Operations Event Management and Service Maps

Openview Service Desk Figure Openview Service Desk Ticketing and Troubleshooting Customer case studies Customer A Genentech A customer s storage controller reboots with an error Tachyon Master Abort. This error occurs when the switch port where the controller host port is connected becomes unstable. The controller Tachyon chip will go into a recovery loop and eventually resets. The controller error log in this case shows warnings and controller reboots. The switch error counters on the ports where the controller hostports are connected show a number of connection errors. Each separate Device Management tool shows their respective errors, Storage Essentials can help to analyze the errors end-to-end from host to storage array. When a switch port is faulty, the host will see LUNS disappearing. The storage array Device Management tool will see port errors, as well as the switches. HP Storage Essentials shows the end-to-end path from the server to the connected storage controller port via the connected switch ports, with errors for the specific devices.

After finding the switch ports where the storage controllers are connected, logging into the switches shows that 2 ports are in faulty state. After replacing 2 broken SFPs on the switches, the ports are back in normal state and the controllers are no longer rebooting. Integrated server and storage administration would help to analyze the problem and pro-actively troubleshoot the SAN environment. The storage administrator does not find any hardware issues with the storage controllers themselves, but the end-to-end analysis shows that the host and storage controller issues are related to an unstable switch port connection. These errors can be followed up in a centralized storage administration environment provided by System Insight Manager integration with Storage Essentials, or by forwarding alerts into a central operations helpdesk level, for instance by Openview Operations integrations as second and third level customer helpdesk environment. HP offers integrated management level for server, storage and operations administrators. Figure Storage Essentials EVA controller event Figure shows the sequence of events that happened at the controller level.

Figure Storage Essentials EVA controller event Figure shows the event details on how the storage controller port has an unstable connection issue. Figure Storage Essentials EVA controller event Figure shows how the storage controller port goes into failed state after failing to respond. Figure Storage Essentials EVA controller event Figure shows how the storage controller has gone offline.

Figure Storage Essentials switch port events Figure shows how the switch ports are showing connected devices going offline. Customer B DHL Storage Essentials is a multi-vendor management tool. When hardware errors occur on non-hp storage systems, alerts can be elevated from the storage administrator level up to the operations administration level. All system and storage management application integrations will send SNMP traps. Integration with Openview Operations offers a customer a detailed configurable SPI integration, with policies that can be setup for event forwarding and customization, event filtering that can be configured, and many other options to avoid receiving message storms. System Requirements For this environment, the following supported versions were configured: EVA Command View 6.0.1 Openview Operations for Windows 7.5 Openview Operations for Unix Openview Service Desk System Insight Manager 5.0 SP1 Storage Essentials 5.1 with or without SP1 Openview Operations Agent for Windows and Unix 7.5 For more information www.hp.com/go/hpsim www.managementsoftware.hp.com www.hp.com/go/ovo

2003-2006 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Itanium is a trademark or registered trademark of Intel Corporation in the U.S. and other countries and is used under license. 418413-001, 04/2006