EView/400i Insight for iseries (AS/400)

Transcription

1 EView/400i Insight for iseries (AS/400) Splunk Integration Installation and Administration Guide Software Version: 7.0 July 2015 Copyright 2015 EView Technology, Inc.

2 Legal Notices Warranty EView Technology makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. EView Technology shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material. Restricted Rights Legend All rights are reserved. No part of this document may be copied, reproduced, or translated to another language without the prior written consent of EView Technology, Inc. The information contained in this material is subject to change without notice. Use, duplication or disclosure by the U.S. Government is subject to restrictions as set forth in subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer Software clause at DFARS for DOD agencies, and subparagraphs (c) (1) and (c) (2) of the Commercial Computer Software Restricted Rights clause at FAR for other agencies. EView Technology, Inc Green Road Raleigh, North Carolina United States of America Copyright Notices Copyright 2015 EView Technology, Inc. No part of this document may be copied, reproduced, or translated into another language without the prior written consent of EView Technology, Inc. The information contained in this material is subject to change without notice. Trademark Notices EView/400 is a registered trademark of EView Technology, Inc. iseries, AS/400 are trademarks of International Business Machines Corporation. Microsoft, Windows is a U.S. registered trademarks of Microsoft Corporation. UNIX is a registered trademark of the Open Group. All other product names are the property of their respective trademark or service mark holders and are hereby acknowledged. 2

3 Contents Table of Contents Concepts... 6 About EView/400i Architecture and Data Flow... 7 Increasing Productivity... 7 What the EView/400i Agent Does... 8 Forwarding iseries Messages... 8 Event and Message Buffering... 8 Splunk Dashboards... 9 EView Dashboard... 9 Installing and De-installing EView/400i Installation Requirements Hardware Requirements Software Requirements Obtaining License Keys Installing EView/400i on a Windows or Linux Server Installation Steps for Windows Installation Steps for Linux Installing EView/400i on the iseries Agent Installing the Library Running the Installation Program Start the EVSBS Subsystem Cleanup of Temporary Files Installing the EView/400i Splunk Application Configuring the Splunk Forwarder Stopping the EVSBS Subsystem

4 Contents De-installing EView/400i To Remove EView/400i Components from the Splunk Forwarding Server To Remove EView/400i from the iseries systems Configuring EView/400i Phase 1: Add iseries Node Configuration Phase 2: Add, Modify, and Distribute Message Queues and Message IDs Configure Message Queues Configure Message ID Filters Message Queue Filters QHST Filters Phase 3: Identify Command Audit Filters Using EView/400i Collecting iseries Messages on the Splunk Forwarding Server Collecting Performance Data Troubleshooting EView/400i General Troubleshooting Use EVSTATUS Command to Verify Status of iseries Agent Specific Troubleshooting Verifying Connectivity and Agent Operation EView/400i Agent Jobs EView/400i Subsystem (EVSBS) Message Text of Audit Journal Entries Audit Journal Type AD (Auditing changes) Audit Journal Type AF (Authority failure) Audit Journal Type AU (Attribute changes) Audit Journal Type CA (Authority changes) Audit Journal Type CD (Command string) Audit Journal Type CO (Create Object) Audit Journal Type CP (User profile changed, created, or restored) Audit Journal Type DO (Delete Operation)

5 Audit Journal Type DS (DST security password reset) Audit Journal Type NA (Network Attribute Change) Audit Journal Type OW (Object ownership changed) Audit Journal Type PA (Program changed to adopt authority) Audit Journal Type PG (Change of an object's primary group) Audit Journal Type PW (Invalid password) Audit Journal Type ST (Use of service tools) Audit Journal Type SV (System value changed) Audit Journal Type VA (Changing an access control list) Audit Journal Type VP (Network password error) Audit Journal Type VU (Changing a network profile) Audit Journal Type ZC (Object accessed (changed)) Audit Journal Type ZR (Object accessed (read)) Performance Collection Metrics Classes Selecting Performance Metrics PERFDATA PERFDATA

6 1 Concepts This chapter describes EView/400i Insight (EView/400) and provides a brief overview of its benefits, architecture, and data flow. 6

7 About EView/400i Architecture and Data Flow EView/400i consists of two main components: the agent component that runs on the iseries (AS/400) server, and the server component that runs on the EView Splunk Collector server. Events and performance data are forwarded from the agent to the EView Splunk Collector and written to a file that is monitored by a standard Splunk forwarder. The EView Splunk Collector sends data to the Splunk server where the EView/400i Splunk app maps data from common event fields. The EView/400i Splunk app contains dashboards to help get you started in viewing iseries event and performance data. Figure 1-1 shows the data flow between the iseries, the EView/400 Splunk Collector and the Splunk server. Figure 1-1: EView/400i Data Flow Increasing Productivity Consolidating the events of mainframes and other systems with Splunk enables you to act proactively and quickly analyze data from all of your enterprise systems. Using this intuitive and cost-effective solution as the central end-user interface provides the basis for enterprise problem analysis. 7

8 What the EView/400i Agent Does The EView/400i agent operates as a subsystem with multiple jobs. iseries messages are collected by the agent from several sources, outlined below. Pre-defined messages filters identify important messages that are then packaged into a common data structure and forwarded via TCP/IP to the Splunk server for processing. Forwarding iseries Messages Messages can include information from the following: System Operator Message Queue (QSYSOPR) Application Message Queues History Log (QHST) System Audit Journal System Performance Data Event and Message Buffering If event, message, or performance data cannot be sent to the EView Splunk Collector for any reason, the EView/400i agent will save or buffer the data until the connection from the EView Splunk Collector is available. This ensures that important data will not be lost. 8

9 Concepts Splunk Dashboards The EView/400i Splunk app contains default dashboards to provide examples of different ways iseries data can be viewed as the information is seamlessly integrated into Splunk. EView Dashboard The EView Performance and Event Dashboard shows performance data and system audit events. Figure 1-2: EView Performance and Event Dashboard 9

10

11 2 Installing and De-installing EView/400i This chapter describes how to install and de-install EView/400i Insight for Splunk (EView/400). EView/400i Insight for Splunk consists of two components. The Client component is installed on a Windows or Linux server where a Splunk forwarder is installed. The Agent component is installed on each iseries (AS/400) operating system partition that will be sending event and performance data. The EView/400i Insight for Splunk is installed first on a Windows or Linux server and includes the Agent software installation file which is transferred to the iseries partitions for installation. 11

12 Installation Requirements This section describes the operating system, hardware, and software requirements for installing EView/400i software. To avoid problems during installation, read this section before you start the installation process. Hardware Requirements EView Splunk Collector - Intel 64-bit architecture - Appropriate Ethernet hardware on the client to communicate via TCP/IP iseries (AS/400) Agent - Appropriate Ethernet hardware on the iseries to allow for TCP/IP communication with the EView Splunk Collector In addition, make sure that the EView Splunk Collector and iseries partitions meet the disk space requirements described in Table 2-1. Table 2-1: Additional Disk-Space Requirements Platform EView Splunk Collector iseries Disk Space 5MB 50MB Software Requirements On the EView Splunk Client: Windows Client: - Microsoft Windows 2008 R2 or later Linux Client: - Linux 64-bit kernel Version or later - Perl Version 5.8 or later - glibc Version 2.7 or later The TCP/IP network protocol stack must be active. All other software requirements are the same as the requirements for a Splunk forwarding server. On the iseries agent: System i OS V5.1 or later The TCP/IP network protocol stack must be active. 12

13 Obtaining License Keys EView/400i requires a license key to be applied to the configuration of each iseries system that will be configured on the EView Splunk Collector. One license is required for each physical iseries system. The same license key may be used for multiple LPARs on the same physical system. Contact EView Technology at or support@eview-tech.com to get the necessary license keys. Be prepared to give the serial number and processor group of the iseries system. The serial number can be found by issuing the DSPSYSVAL QSRLNBR command on the iseries system. The processor group can be found by issuing the WRKLICINF OUTPUT(*) command on the iseries. 13

14 Installing EView/400i on a Windows or Linux Server The EView/400i installation program is run as an executable on a Windows server or installed using the Linux RPM install process on a Linux server. Installation Steps for Windows 1. Copy the EView/400i Insight installation executable to the server where it is to be installed. 2. Double-click Eview400InsightInstall.exe. 3. The installation process copies the necessary files to the Splunk forwarding server in the directory path you specify. The default path for EView/400i files is: \Program Files\EView Technology\EView 400\ Installation Steps for Linux 1. Copy the EView/400i Insight rpm file to the Splunk forwarding server where it will be installed. 2. Run the Linux rpm command: rpm --install /tmp/eview400insight-7-0.x86_64.rpm where /tmp is the directory where the rpm file was saved. After the rpm command is run, the vp400conf service will start which will allow access to the web browser configuration application. (See Phase 1: Add iseries Node Configuration on page 23.) 14

15 Installing and De-installing EView/400i Installing EView/400i on the iseries Agent This section explains how to start the EView/400i installation process on the iseries agent using the following steps: 1. Library installation 2. Running the Install Program 3. Start the EVSBS Subsystem 4. Cleanup of Temporary Files Installing the Library Use the EDTLIBL command to verify that the EVIEW library is not in your library list on the iseries agent. Follow these steps to load the agent components of EView/400i: 1. Sign on to the iseries system as QSECOFR or other user with *ALLOBJ authority. 2. Create a temporary save file named EVREL70 in any available library (e.g., QGPL) to receive the installation save file: CRTSAVF FILE(libname/EVREL70) 3. On the Splunk forwarding server, change directory to the as400 directory: - On Windows: cd \Program Files\EView Technology\EView 400\as400 - On Linux: cd /opt/ov/vp400/as400 then start an ftp session to the iseries system. Set the file type to binary, then change directory to the library name of the save file created in Step 2. Use the put command to place the library on the iseries agent. # cd /opt/ov/vp400/as400 # ftp iseriesname User: qsecofr Password: **** ftp> bin ftp> cd libname ftp> put EVREL70.SAVF ftp> quit 4. Restore the EView/400i library on the iseries (a temporary library named EVREL70 will be created): RSTLIB SAVLIB(EVREL70) DEV(*SAVF) SAVF(libname/EVREL70) RSTLIB(EVREL70) 15

16 Running the Installation Program From an iseries command line, enter the following command to create the EView/400i runtime library, EVIEW: EVREL70/EVINSTALL Press F4 to see the installation options, or use the defaults described below: Table 3-2 EVINSTALL Options Parameter Keyword Default Description MMS Port MMSPORT 9000 The TCP/IP port number which will be opened and listened on for connections from the MMS process on the forwarding server. Enter any unused port number between 1024 and This number will be ignored if the installation is upgrading a previous EView/400i version. CS Port CSPORT 9001 The TCP/IP port number which will be opened and listened on for connections from the CSS process on the forwarding server. Enter any unused port number between 1024 and This number will be ignored if the installation is upgrading a previous EView/400i version. HCI Port HCIPORT 9002 The TCP/IP port number which will be used for inter-process communications by jobs in the EVIEW subsystem. Enter any unused port number between 1024 and This number will be ignored if the installation is upgrading a previous EView/400i version. Backup Library BACKUPLIB QGPL The name of the library where a backup of the current EVIEW library will be saved with a savefile name EVIEW7SAVE. This parameter will only be used if the installation is upgrading a previous EView/400i version. Use *NONE to skip the creation of a backup savefile. 16

17 Installing and De-installing EView/400i Start the EVSBS Subsystem Start the EVSBS subsystem using one of the following commands: CALL EVIEW/EVINIT or: STRSBS EVIEW/EVSBS The EVSBS subsystem will start using either the TCP/IP port numbers specified in the EVINSTALL, or the existing defined port numbers if the installation is an upgrade from a previous EView/400i version. Optional PARM values are available for the EVINIT command: ALL CLEARQ TEST VERSION jobname Start all jobs in the subsystem that have been configured. This is the default option. Clear any buffered messages from the EView message queues before starting the subsystem jobs. Instead of starting the jobs, EVINIT will display the SBMJOB commands that would be used to start the jobs. This may be useful to verify that the jobs are being started with the desired options. Display the version of the installed agent software. Start specific job(s) in the subsystem. If a job has fallen into a Message Wait status, use ENDJOB OPTION(*IMMED) to stop the individual job, then restart it by specifying the specific process name in the PARM when calling EVINIT. Job names are listed in Appendix A, or use the EVIEW/EVSTATUS PARM('JOBS') command to view which jobs are not running that should be. Examples: To clear the agent s internal data queues before starting the EVSBS agent subsystem: CALL EVIEW/EVINIT PARM(CLEARQ) To start only the EVSHSTPROC, EVSTCPPROC, and EVTCTLPROC jobs: CALL EVIEW/EVINIT PARM('EVSHSTPROC EVSTCPPROC EVTCTLPROC') (Specify individual job names only when the EVSBS subsystem is already running.) Cleanup of Temporary Files Delete the temporary installation library and save file: DLTLIB LIB(EVREL70) 17

18 18 DLTF FILE(libname/EVREL70)

19 Installing and De-installing EView/400i Installing the EView/400i Splunk Application Download the.spl file from the Splunk apps web page at Alternatively, use the eview400i_insight.spl file from the Splunk forwarding server, found in the following directory: - On Windows: \Program Files\EView Technology\EView 400\splunkapp - On Linux: /opt/ov/vp400/splunkapp On the Splunk server GUI, go to the Apps page and click the Install app from file button. On the Upload app page, identify the location of the EView application file: Figure 1-1: Upload App 19

20 Configuring the Splunk Forwarder Refer to the Splunk documentation to install the forwarder software on the Splunk forwarding server and connect it to the server/indexer. After the forwarder is installed, follow these steps to add EView/400i to the list of sources for the forwarder: 1. Edit the inputs.conf file in the following directory: - On Windows: \Program Files\Splunk\etc\system\local\ - On Linux: /opt/splunk/etc/system/local/ (Create a new inputs.conf file in this directory if it does not already exist.) 2. Add the following four lines to the end of the inputs.conf and save the file: On Linux: [monitor:///var/opt/ov/log/vp400/ev400.insight.*.log] host_regex = ev400\.splunk\.(.+)\.log sourcetype = eview-iseries disabled = false On Windows: [monitor://c:\program Files\EView Technology\EView 400\log\ev400.insight.*.log] host_regex = ev400\.splunk\.(.+)\.log sourcetype = eview-iseries disabled = false (If the EView path was changed during the installation, modify the first line to the new path.) 3. Restart the Splunk forwarder. On Linux: /opt/splunk/bin/splunk restart On Windows: Restart the Splunk forwarder service splunkd. 20

21 Stopping the EVSBS Subsystem To terminate a running EView/400i subsystem on the iseries agent, use the command: ENDSBS EVSBS *IMMED The EVSBS subsystem must be ended prior to executing any save commands that would allocate an EView/400i object, such as when performing a backup. De-installing EView/400i This section describes how to remove EView/400i software from the following: Splunk forwarding server iseries managed nodes To Remove EView/400i Components from the Splunk Forwarding Server On Windows: use the Add/Remove Programs utility from Windows Control Panel to remove EView/390z files and registry entries. On Linux: use the rpm command: rpm --erase EView400SPLUNK-7-0.x86_64 To Remove EView/400i from the iseries systems To remove EView/400i from the managed nodes, follow these steps: 1. Stop the EView/400i subsystem using the OS/400 command: ENDSBS EVSBS *IMMED 2. Enter the following commands to delete the EVIEW library from the iseries system: CLROUTQ EVIEW/EVCMD CLROUTQ EVIEW/EVTRACE CLROUTQ EVIEW/EVHSTOQ DLTLIB LIB(EVIEW) 3. Enter the following command to delete the EVUSER user profile: DLTUSRPRF USRPRF(EVUSER) 21

22 3 Configuring EView/400i This chapter describes how to configure and start the EView/400i component on the Splunk forwarding server. 22

23 Phase 1: Add iseries Node Configuration New iseries nodes to be monitored by Splunk must first be configured using the EView/400i web configuration interface. The configurator is launched using a web browser and URL constructed as follows: where proxyserver is the hostname or IP address of the Splunk forwarding server where the EView/400i software was installed is the default port number used by the web configuration interface. If port 9850 is not available the default port number can be changed by editing the vp400info file in the EView/400i configuration directory and changing parameter EV400_CONFIG_PORT: On Windows: On Linux: \Program Files\EView Technology\EView 400\parm\ev400info /etc/opt/ov/share/conf/vp400/vp400info then restart the configurator service: On Windows: On Linux: Stop and restart the EView/400i Configurator service service vp400conf stop service vp400conf start The EView/400i Node Configuration screen is used to add a new iseries LPAR to be monitored by Splunk. (Splunk views LPARs as separate nodes, even if they exist on the same physical box.) Figure 3-1 EView/400i Configurator To add iseries nodes follow these steps: 1. Start the EView/400i configuration interface from a web browser. 2. Click on the Add Node button and enter the iseries LPAR s fully qualified hostname as defined in your DNS. 23

24 3. Highlight the new node and click Edit Node to configure the EView/400i server parameters. Update the EV400_LICKEY parameter with the license key supplied by EView Technology for this system. You can accept the default values created (recommended) or provide custom values for the configuration parameters. The node parameters are: Table 3-1 EView/400i Node Parameters Parameter Description Valid Values Default Value EV400_ADDMSG_FIELDS Indicates whether EView/400i will send the Program Name and Message Type fields in the messages that are sent to the server. YES The EView/400i message server will send the Program Name and Message Type fields in its messages to the server. These fields were added in the EView/400i Version 2.0 and will need to be accommodated in any existing template conditions that were written for Version 1.0 of the product. NO Use this option if you are using message template conditions from EView/400i Version 1.0 and do not wish to modify those existing templates to utilize the new fields. YES EV400_AS400_ADDR Fully qualified network name of the iseries system where the EView/400i agent component is installed. Name of iseries managed node. None EV400_AS400_BIND_ADDR Address on the agent that the EVSBS IPV4 dotted decimal address in the subsystem should bind to when opening format nnn.nnn.nnn.nnn. The value its TCP/IP listening ports (useful when must be a defined address on the the iseries system iseries has multiple IP addresses defined) (the INADDR_ANY default) EV400_AS400_CMD_PORT TCP/IP port number assigned to the EView/400i Command Server process. Any unused port number on the iseries agent between 1024 and EV400_AS400_CMD_RSP_PORT TCP/IP port number assigned for communication between the EView/400i Message Server process and Command Server process Any unused port number on the forwarding server between 1025 and * EV400_AS400_MSG_PORT TCP/IP port number assigned to the EView/400i Master Message Server process. Any unused port number on the iseries agent between 1024 and

25 Parameter Description Valid Values Default Value EV400_AS400_SERV_ADDR EV400_AS400_SERVER_PORT EV400_CMD_CLIENT_PORT Address (or address range) of the IPV4 dotted decimal address in the Any forwarding server(s) that are allowed to format nnn.nnn.nnn.nnn, optionally address may connect to the iseries agent. Use a / followed connect to the followed by a CIDR prefix or subnet by a slash and either a dotted decimal listening mask address representation of a subnet EView/400i to specify a range of allowed addresses. mask ports or a number (0-32) representing the number of bit positions to use for the mask. A TCP/IP port number reserved for inter-process communications on the iseries agent. Any unused port number on the iseries agent between 1025 and A TCP/IP port number used by the Any unused port number on the Command Server process to forwarding server between 1025 and communicate with the Master Message Server process. This port number must be unique on the forwarding server where the Command Server and Master Message Server processes are running * EV400_CMD_SERVER_ADDR The name of the forwarding server where the Command Server process is to run. A DNS-recognized server name. The forwarding server name EV400_CMD_TIMEOUT The amount of time to wait for an An integer greater than or iseries command response (in seconds). equal to 1 (second). 30 EV400_LICKEY License key for the managed node. To obtain a license key, contact EView Technology support at support@eview-tech.com. (See page 13.) Valid license key None EV400_MON_AUDJRNL EV400_MONITOR_QHST A list of two-character entry types from AD,AF,AU,CA,CD,CO,CP,DO,DS, the QAUDJRN that should be NA,OW,PA,PG,PW,ST,SV,VA,VP, forwarded VU,ZC,ZR from the iseries agent. Entry types are ALL All of the above separated by commas. See iseries NONE None of the above documentation (such as the iseries Security Reference) or Appendix B for descriptions of journal entry types. Indicates whether the EView/400i YES or NO agent should monitor for messages that are sent to the system QHST history log. If set to YES, then verify that the EV400_QHST_MON_FREQ field is greater than 0. NONE YES EV400_MON_RESOURCES Indicates whether the EView/400i agent should monitor the status of iseries resources (lines, controllers, and devices). This function is not used in EView/400i for Windows and should YES or NO NO 25

26 Parameter Description Valid Values Default Value always be set to "NO". EV400_MSG_DISTRIB EV400_MSG_SERVER_ADDR Should the iseries agent send its collected messages to all servers that are in contact with it? (If NO, then specify in the EV400_PRIMARY_SERVER field which server is the primary recipient of messages.) YES Send unsolicited iseries messages to all EView/400i servers that are in contact with this agent. NO Send unsolicited messages only to the primary server. The name of the forwarding server A DNS-recognized server name. where the Master Message Server process is to run. YES The forwarding server name. EV400_PATH The EView/400i installation directory on the forwarding server. EView/400i home directory Windows: \Program Files\EView Technology\ EView 400\ Linux: /opt/ov/vp400 EV400_PERF1 EV400_PERF1_INT EV400_PERF2 EV400_PERF2_INT EV400_PRIMARY_SERVER Specifies whether the performance gathering function will be activated on the iseries agent to gather the data for performance group 1. See Appendix C for the list of metrics collected in group 1. YES Activate the performance gathering function on the iseries agent. NO Do not activate performance data gathering for group 1. The interval, in minutes, at which group An integer greater than or equal 1 performance data is collected on the to 1 (minute). iseries agent and sent to the forwarding server. This field is only needed if EV400_PERF1 is set to "YES". Specifies whether the performance gathering function will be activated on the iseries agent to gather the data for performance group 2. See Appendix C for the list of metrics collected in group 2. YES Activate the performance gathering function on the iseries agent. NO Do not activate performance data gathering for group 2. The interval, in minutes, at which group An integer greater than or equal 2 performance data is collected on the to 1 (minutes). iseries agent and sent to the forwarding server. This field is only needed if EV400_PERF2 is set to "YES". The fully qualified name of the primary An EView/400i forwarding server forwarding server to receive messages name from this agent. Although multiple servers may be connected to the iseries agent at one time, only the server named here will receive unsolicited iseries messages This field is only necessary when the NO 5 NO 30 null 26

27 Parameter Description Valid Values Default Value EV400_MSG_DISTRIB parameter is NO. EV400_QHST_MON_FREQ EV400_WORK_AREA EV400_VP400CS_TRACE EV400_VP400MMS_TRACE EV400_VP400HOSTCMD_TRACE EV400_EVCMSG_TRACE EV400_EVCHCI_TRACE EV400_EVC050_TRACE Frequency (in seconds) that the An integer greater than or EView/400i agent collects new messages equal to 1 (seconds) from the system QHST history log. This field is only necessary when the EV400_MONITOR_QHST parameter is YES. Specifies where EView/400i places Any existing directory on the temporary work files on the forwarding forwarding server server. Set tracing level for the command server 0 - No tracing output enabled (ev400cs on Windows, vp400cs on general program trace enabled Linux) internal tracing enabled program detail tracing enabled Multiple values can be added together warning messages enabled in hexadecimal error tracing enabled dump output enabled loop tracing enabled verify tracing enabled Set tracing level for the master message 0 - No tracing output enabled server (ev400mms on Windows, general program trace enabled vp400mms on Linux). Multiple values internal tracing enabled program detail tracing enabled can be added together in hexadecimal warning messages enabled error tracing enabled dump output enabled loop tracing enabled verify tracing enabled log messages sent to Splunk log performance records Set tracing level for the host command client. Multiple values can be added together in hexadecimal. Set tracing level for the agent message TCP/IP task (EVCMSG). Multiple values can be added together in hexadecimal. Set tracing level for the agent message transfer process (EVCHCI) 0 - No tracing output enabled general program trace enabled internal tracing enabled program detail tracing enabled warning messages enabled error tracing enabled dump output enabled loop tracing enabled verify tracing enabled 0 - No tracing output enabled general program trace enabled internal tracing enabled program detail tracing enabled warning messages enabled error tracing enabled dump output enabled loop tracing enabled verify tracing enabled 0 - disables tracing 1 - enables tracing Set tracing level for the agent command 0 - disables tracing processor (EVC050) 1 - enables tracing 30 /var/opt/ov/ share/tmp/vp EV400_EVCQSCAN_TRACE Set tracing level for the agent message queue monitor (EVCQSCAN) 0 - disables tracing 1 - enables tracing 0 27

28 Parameter Description Valid Values Default Value EV400_EVPERFM_TRACE Set tracing level for the agent performance monitor process (EVPERFM) 0 - disables tracing 1 - enables tracing 0 EV400_EVCCTL_TRACE Set tracing level for the API interface process (EVCCTL) 0 - disables tracing 1 - enables tracing 0 EV400_EVC070_TRACE Set tracing level for the agent resource monitor (EVC070) 0 - disables tracing 1 - enables tracing 0 EV400_EVCCMD_TRACE Set tracing level for the agent command 0 - disables tracing TCP/IP process (EVCCMD) 1 - enables tracing 0 EV400_EVHSTPGM_TRACE Set tracing level for the agent history log (QHST) monitor 0 - disables tracing 1 - enables tracing 0 EV400_VP400MMS_LOGSIZE EV400_VP400CS_LOGSIZE Set the maximum log size in 1K increments for the master message server (ev400mms) Set the maximum log size in 1K increments for the command server (ev400cs) (kilobytes) (kilobytes) 3000 EV400_CMDRSP_CODEPAGE Set the code page to be used for converting command responses. Any codepage supported by the forwarding server such as 1252 Latin I 932 Japanese Shift-JIS 936 Simplified Chinese 949 Korean A value of UTF-8 indicates that command output is not converted using any codepage. Set an alternate code page to be used for Any valid code page, but EV400_CMDRSP_ALT_CODEPAGE converting command responses when in most cases the default value of EV400_CMDRSP_CODEPAGE is 437 would be used. set to UTF-8. This parameter is only used with the ev400hostcmd option 81. If EV400_CMDRSP_CODEPAGE is not set to UTF-8, this parameter is ignored. UTF EV400_NLS_CCSID Set the CCSID for the language library that is being used as the subsystem library for the EView/400i agent subsystem. Any CCSID supported on i5os (OS/400). Some typical values are: QSYS2924 English 37 QSYS2928 French 297 QSYS2929 German 273 QSYS2931 Spanish 284 QSYS2932 Italian 280 QSYS2962 Japanese 5026 QSYS2986 Korean 933 QSYS2989 Simplified Chinese * This port number will be incremented automatically for new nodes that are added so that the port numbers remain unique on the server. 28

29 4. Save the parameters for this agent. The Node Configuration program will save the parameters locally on the forwarding server. 5. Select any nodes in the list of defined nodes that have the Distributed? field marked as No and click the [Distribute...] button to send the configuration parameters to the iseries agents. The EVSBS subsystem on the iseries system must be running to accept the parameters. If the edited parameters result in a change to the operation of the agent, restart the EVSBS subsystem. 6. Click the [Start] button to start the EView/400i server processes for the iseries node. Phase 2: Add, Modify, and Distribute Message Queues and Message IDs iseries messages can be captured from any message queue or the QHST message log. This section explains how to identify which queues are to be monitored and which messages should be captured and passed from the EView/400i agent to the Splunk forwarding server. Configure Message Queues 1. Start the Message Queue Configuration utility from the EView/400i Configurator (Figure 3-1) by clicking on the Message Queue Configuration link. Figure 3-2: Message Queue Configurator 2. To change the message queues being monitored, add a new configuration group using the [New] button or edit an existing group using the [Edit] button. 29

30 Figure 3-3 Editing a Message Queue Group 3. The QSYSOPR/QSYS queue is listed by default in a group. Use the [New Row] button to add another line for additional queues to be added to this group. To delete a listed queue, check the trash can icon to the right of the line. The options for each queue are: In the Message Queue field, enter the name of the message queue to be monitored. In the Library field, enter the name of the library where the message queue resides. Set the Filter option to one of the following: YES: if the message ID filters should be applied to messages coming from this queue, restricting which messages will be forwarded to the server. NO: allow messages to be passed on to the server regardless of their message ID. SEV: allow any non-inquiry messages with a severity equal to or greater than the Min Sev. field to be forwarded to the server regardless of the message ID. Messages with a severity less than the Min Sev. value will be forwarded only if the message ID is in the message ID filter table. Set the Mode option to one of the following: BREAK: to allow EView/400i to set the queue in *BREAK mode. EView/400i provides a break message-handling program that will be called each time a new message is written to the queue. Break Mode advantage: instant processing of incoming messages. 30

31 SCAN: to have EView/400i scan the queue on the interval (by default, every 5 seconds) to check for new messages. Scan Mode advantage: does not require a lock on the message queue and can co-exist with other message queue monitoring programs. Set the Min Sev. field to a numeric value 0-99 indicating the necessary minimum severity of an incoming message. Messages with a lower severity will not be passed on to the server, even if matched to a message ID filter. Enter 0 to allow all messages to be processed, regardless of severity. In the Age Limit field, enter a time limit (in seconds) of how old a message can be and still be passed on to the forwarding server. This field is only used for queues that are monitored with the "Scan" mode option (see above). This is useful during startup of the subsystem on the iseries agent. When the subsystem is started for the first time (or if it has been brought down for any length of time), the Age Limit prevents the agent from sending a flood of old unnecessary messages to the server. If the Inquiry field is set to "Yes" then all messages in that queue with a Message Type of Inquiry (messages that ask for a reply) will be forwarded to the server, regardless of the message ID if the Filter option is set to "Yes". 4. Click the [Confirm] button when all message queues are added to the configuration group. 5. Click the [Assign] button to assign queue configuration groups to iseries agents. The same configuration group may be assigned to multiple agents. 6. Select a queue configuration and click the [Distribute] button to send the list of monitored queues to the iseries agent. The EView/400i EVSBS subsystem must be running on the agent at the time of the distribution. When the EView/400i agent subsystem is running, it will begin monitoring message queues defined with Scan mode immediately after the distribution is completed. Queues defined with Break mode monitoring will begin monitoring after the next time the EVSBS subsystem is restarted. Configure Message ID Filters Message ID filters restrict the number of messages that are sent from the iseries agent to the forwarding server and save the server from receiving a flood of unnecessary messages. Each iseries agent has two message filters, one for message queues and one for the QHST message log. Message Queue Filters Start the Message Queue Filters application from the EView/400i Configurator (Figure 3-1) by clicking on the [Message Queue Filters] link. Figure 3-4: Message Queue Filters 31

32 32 1. To change the list of message IDs that are sent to the forwarding server, add a new filter group using the [New] button or edit an existing one using the [Edit] button. New filters may also be created by copying an existing filter or the supplied default filter (default.msg.filter) by selecting an existing filter and using the [Copy] button.

33 Figure 3-5: Editing a Message Queue Filter 2. Enter new message IDs to the list in the open field and click the [Add Msg ID] button. To delete from the list, click the message ID(s) to remove and click the [Delete Msg ID] button. Message IDs must be no more than seven alphanumeric characters, but any message ID entered can contain the special period character (.) to indicate that any character in that position should match. If the message ID is terminated with an asterisk (*), matching will only occur on characters preceding the asterisk. See the following examples: Table 0-2: Message Filter Examples To forward the following messages: All messages * ABC1234 All messages beginning with ABC Any 7-character message beginning with ABC and ending with 9 Enter the following in the list of Message IDs: ABC1234 ABC* ABC Click [Confirm] button when all message IDs are added to the filter group. 4. Click the [Assign] button to assign filter groups to iseries agents. The same filter group may be assigned to multiple agents. 5. Select a filter group name and click the [Distribute] button to send the list of message IDs to the iseries agent. The EView/400i EVSBS subsystem must be running on the agent at the time of the distribution. The EView/400i agent subsystem will begin monitoring with the new message ID filters immediately after the distribution is completed. 33

34 QHST Filters Start the QHST Filters utility from the EView/400i Configurator by clicking on the [QHST Filters] button. 1. To change the list of message IDs that are sent to the server, add a new filter group using the [New] button or edit an existing one using the [Edit] button. 2. Enter new message IDs to the list in the open field and click the [Add Msg ID] button. To delete from the list, click the message ID(s) to remove and click the [Delete Msg ID] button. 3. Click the [Confirm] button when all message IDs are added to the filter group. 4. Click the [Assign] button to assign filter groups to iseries agents. The same filter group may be assigned to multiple agents. 5. Select a filter group name and click the [Distribute] button to send the list of message IDs to the iseries agent. The EView/400i EVSBS subsystem must be running on the agent at the time of the distribution. The EView/400i agent subsystem will begin monitoring with the new message ID filters immediately after the distribution is completed. Phase 3: Identify Command Audit Filters The Command Audit Filters work with the iseries QAUDJRN audit journal to determine which audit entries of type CD (Command) will be forwarded to the server. If an iseries user s profile is set up (using CHGUSRAUD) to journalize the user s issued commands, the Command Audit Filters can be used to reduce the number of journal entries that are forwarded to the server. Note that this section is only necessary if the CD value is specified in the EV400_MON_AUDJRNL parameter for this node. 1. To change the list of commands that are sent to the forwarding server, add a new filter group using the [New] button or edit an existing one using the [Edit] button. 2. Enter new commands to the list in the open field and click the [Add Command] button. To delete from the list, click the command(s) to remove and click the [Delete Command] button. 3. Click the [Save and Close] button when all commands are added to the filter group. 34

35 Figure 3-6: Editing the Command Audit Filters 4. Click the [Assign] button to assign filter groups to iseries agents. The same filter group may be assigned to multiple agents. 5. Select a filter group name and click the [Distribute] button to send the list of commands to the iseries agent. The EView/400i EVSBS subsystem must be running on the agent at the time of the distribution. See Appendix B for the displayed format of the CD and other audit journal command types. 35

36 Phase 3: Identify Command Audit Filters 4 Using EView/400i This chapter describes how to use EView/400i Insight to capture mainframe messages and forward them to Splunk. 36

37 Using EView/400i Collecting iseries Messages on the Splunk Forwarding Server The EView/400i component on the Splunk forwarding server writes mainframe messages to a log file in one of the following directories: on Windows: on Linux: \Program Files\EView Technology\EView 400\log /var/opt/ov/log/vp400 The log s file name will include the name of the iseries system that is being monitored. Each line of the log file will contain a timestamp, source prefix, and message text. The source prefix indicates where the message was generated on the iseries: *OS400 MSG Message from an iseries message queue or the QHST history log or System Audit Journal *PERFDATA1 Performance data from Group 1 *PERFDATA2 Performance data from group 2 These source prefixes will be interpreted by the Splunk server when displaying the messages. Collecting Performance Data If the optional performance job EVPERFPROC is running on the iseries agent, *PERFDATA1 and/or *PERFDATA2 lines will be sent to the Splunk server for analysis at the desired interval. See Appendix C for the description of metrics collected. 37

38 Collecting Performance Data 5 Troubleshooting EView/400i This chapter describes how to troubleshoot problems with EView/400i. 38

39 Troubleshooting EView/400i General Troubleshooting Before you troubleshoot a particular problem you run into when using EView/400i, you should verify that your EView/400i environment is correctly installed and configured. Correct installation and configuration of EView/400i ensures, among other things, that messages are processed correctly: Message Capture Messages are collected by the EView/400i agent from the several sources on the iseries system. Message Transmission Messages are sent to the EView/400i server component on the Splunk forwarding server. Use EVSTATUS Command to Verify Status of iseries Agent On the iseries agent, use the command EVIEW/EVSTATUS to collect the status of the several components of the EView/400i agent and their interaction with the iseries system. The command is called from an iseries (5250) terminal. The format is: EVIEW/EVSTATUS PARM('options') OUTPUT(outoption) where: options outoption One or more of the following, separated by spaces: VER EView/400i version information CONF Current distributed configuration files JOBS Status of EVSBS jobs TCP Defined TCP/IP ports and current status DQS Data queues status AUD System QAUDLVL vs. EView/400i audit options USP Defined user spaces SYS iseries system information ALL All of the above (Default)? or HELP Display help options One of: * For output to a terminal *PRINT For output to the user's print queue (Default) Example call: EVIEW/EVSTATUS PARM('JOBS TCP SYS') OUTPUT(*) 39

40 Specific Troubleshooting Browse the output text of this command and look for NOTE or WARNING messages that may indicate how to resolve outstanding problems. Retain a copy of the output for possible transmission to support personnel. Specific Troubleshooting This section explains how to solve specific problems you may encounter when using EView/400i. Verifying Connectivity and Agent Operation Symptom No apparent communication between the Splunk forwarding server and the iseries agent. Solution To verify the correct operation of the server and agent components, use the following steps: On the forwarding server: 1. Start the EView/400i web configurator interface. Verify that all processes are running for that agent. If a node's Command Server is running but the Master Message Server is not, this is usually due to an incorrect license key. Check the ev400mms log file for this error (step 3 below). 2. Check the status of the TCP/IP ports used to connect to the agent. For example, if the default ports (9000 and 9001) are used, issue the command netstat a and look for ports 9000 and 9001 to have a status of "Established". 3. Check for errors in the ev400mms.as400name.log and ev400cs.as400name.log files On Windows: in the \Program Files\EView Technology\EView 400\log directory On Linux: in the /var/opt/ov/log/vp400 directory. On the iseries managed node: 1. Enter the EVIEW/EVSTATUS command as described on page 39. Look for any Note or Warning messages in the output which may indicate a problem. 40

41 Troubleshooting EView/400i 2. Issue the command WRKACTJOB SBS(EVSBS) The following six jobs should be listed in an active (not "Message Wait") status: EVACMDPROC PGM-EVCCMD EVCCTLPROC PGM-EVCCTL EVSCMDPROC PGM-EVC050 EVSMSGPROC PGM-EVC010 EVSTCPPROC PGM-EVCHCI EVTCTLPROC PGM-EVCMSG If the QHST monitoring option was selected in the iseries node's configuration (EV400_MONITOR_QHST parameter is "YES"), then verify the additional job is active: EVSHSTPROC PGM-EVHSTCL If the performance monitoring option was selected in the iseries node's configuration (EV400_PERF1 and/or EV400_PERF2 parameter is "YES"), then verify the additional job is active: EVPERFPROC PGM-EVPERFM Check the agent message queue for any errors that may have been issued: DSPMSG EVIEW/EVLOGQ 3. Check the agent trace files for any error output. The trace files are in the EVTRACE output queue of the EVIEW library: WRKOUTQ EVIEW/EVTRACE 4. Check the status of the TCP/IP ports used by the agent using the command NETSTAT *CNN If the forwarding server processes are connected, the ports configured in parameters EV400_AS400_MSG_PORT and EV400_AS400_CMD_PORT (9000 and 9001 by default) should show as "Established". It is normal for these two ports to also be in a "Listen" state. The port configured in parameter EV400_AS400_SERVER_PORT must be "Established" before any messages can be sent to the forwarding server. 5. Check the condition of the agent data queues. The agent uses several data queues to store requests and messages. Data queue objects may become damaged due to unexpected interruption or system errors, which can cause agent jobs to fail. Issue the following commands to check the data queues: ADDLIBLE EVIEW EVIEW/DDQ EVIEW/EVSENDQ EVIEW/DDQ EVIEW/EVAPIQ EVIEW/DDQ EVIEW/EVCMDQ EVIEW/DDQ EVIEW/EVMRSPQ If a data queue has been damaged, an exception message will be generated when issuing the DDQ command for that queue. If the data queue properties are displayed, verify that the maximum entry length is not zero, which is another indication of a damaged data queue. 41

42 Specific Troubleshooting A EView/400i Agent Jobs This appendix describes the various jobs that run under the EVSBS subsystem on the iseries. 42

43 EView/400i Agent Jobs EView/400i Subsystem (EVSBS) The jobs that execute in the EVSBS Subsystem: Job Name Program Description EVACMDPROC EVCCMD Establishes the TCP/IP socket for bidirectional command and response link. EVCCTLPROC EVCCTL Controls the processing of pre-defined API's used in command processing. EVMSGQMON EVCQSCAN Monitors message queues configured for SCAN mode monitoring. EVPERFPROC EVPERFM Gathers performance data. EVSCMDPROC EVC050 Executes the command processor. EVSMSGPROC EVC010 Message queue allocation and message processing. EVSHSTPROC EVHSTCL Extracts messages at a configured time sequence from the QHST message queue depending on the message ID's added to the filter file. EVSRSCPROC EVC070 Monitors status changes on discovered resources at a configured time sequence. EVSTCPPROC EVCHCI Receives and forwards all processed messages, commands, and API output. Manages a central data queue that allows for message buffering in case the TCP/IP connection to the forwarding server is lost. EVTCTLPROC EVCMSG Controls multiple connectivity between the forwarding server(s) and the EView/400i agent. EVAUDJRNL RCVJRNE The RCVJRNE exit which collects audit records from the QAUDJRN journal. 43

44 44

45 Message Text of Audit Journal Entries B Message Text of Audit Journal Entries This appendix describes how iseries audit records received from the QAUDJRN will be presented to the Splunk forwarding server. All journal messages begin with an AUD0000 message ID header. 45

46 Audit Journal Type AD (Auditing changes) (AD) {cmdname Undefined} command, Object: objname/libname Type: objtype Value: audval Level: {actlvl[,actlvl...] NONE} [DLO Object: dloobj] where: cmdname The command which triggered this audit entry, one of: CHGDLOAUD CHGAUD CHGATTR CHGUSRAUD objname The name of the object for which auditing was changed. libname The name of the library of the object. objtype The type of object. audval The audit value specified in the command. If the scan attribute was changed using the CHGATR command, audval contains the scan attribute value. actlvl The level of activity that is audited for objname. dloobj The DLO object, if one exists. Sample Message: AUD0000 (AD) CHGUSRAUD command, Object: USER1/QSYS Type: *USRPRF Value: *ALL Level: *CMD,*CREATE,*DELETE Audit Journal Type AF (Authority failure) (AF) failuretext [Validation Error Action: actiontext] [(violationcode) violationtext] Object: objname[/libname] [Type: objtype] Job Name: jobname User Profile: usrprf where: failuretext Description of the authority failure, one of: Not authorized to object Restricted instruction Validation failure: Use of unsupported interface Storage protection error ICAPI authorization error ICAPI authentication error Scan exit program action: System Java inheritence not allowed Submit job profile error Profile token not regenerable Optical object authority failure Profile swap error Hardware protection error Default sign-on attempt 46