Enterprise Knowledge Platform

Transcription

1 Enterprise Knowledge Platform EKP Status Monitor Guide 2.1

2 Document Information Document ID: EN150 Document title: EKP Status Monitor Guide Version: 2.1 Document date: 14 April 2009 This document may be revised from time to time. Please check NetDimensions Support site at for updates to this and other documents or send an to to request the most recent version. Please report any errors or feedback with this document by sending an to Copyright Information Copyright by NetDimensions Ltd. All Rights Reserved. Information in this document is subject to change without notice. The software described herein is furnished under a license agreement, and it may be copied only in accordance with the terms of that agreement. No part of this publication may be reproduced, transmitted, or translated in any form or by any means without the prior written permission of NetDimensions Ltd. All company and product names used herein may be trademarks or registered trademarks of their respective companies unless stated otherwise. How to Contact NetDimensions Support US toll-free number [email protected] General Enquiries [email protected]

3 Table of Contents Description... 1 About This Guide... 1 If You Need Help... 1 Operations... 3 Requirements Configuration STATUSFORMAT1 Data Reported Alert Criteria Red Alert Interface Appendix A: Sample sitelist.properties File... 21

4 Description About This Guide This document provides detail instructions for setting up the EKP Status Monitor (ESM), a web-based application for monitoring multiple EKP sites. The ESM allows organizations than manage multiple instances of EKP (for example, hosting services) to have an automated and proactive means of monitoring activity on each of the instances, and reporting the information back in a consolidated, graphical format. ESM is modeled around tools commonly used to monitor network nodes, but its behavior and analysis capabilities are targeted specifically to EKP installations. The ESM works as follows: 1. ESM is installed under Tomcat (or equivalent) on some server; 2. Sites to be monitored are put into a configuration file (sitelist.properties), telling ESM basically how to contact the sites and what parameters to use for alert conditions; 3. ESM will periodically poll the configured sites and analyze the returned status information. 4. A browser session to the ESM (or multiple sessions) at some remote PC is started to report status conditions to a human operator. The browser will automatically update the display every n minutes (default is 5 minutes) to provide a visual guide as to over site/server health. 5. EKP site error conditions will result in browser icons flashing red or yellow, and if the ESM service is monitored by a support call agency, support calls are also triggered. Details of operations and setup are discussed in the remainder of this document. If You Need Help If you cannot resolve a configuration problem using this guide or the online help, or if you should have any queries related to the technology employed within EKP, your first line of contact should be as described in your Technical Support Contract. 1

5 For other queries, or if you are not sure whom to contact, NetDimensions Ltd. may be contacted at Please also refer to the support section of the NetDimensions web site at for the latest information regarding various services. 2

6 Operations The EKP Status Monitor (ESM) is a server-side program that periodically polls a set of EKP sites to verify that they are functioning correctly. Current activity statistics are returned to the monitor that will then display the results in either a server summary screen or a site-by-site detail information screen. In addition, activity data for the previous 300 polling periods (typically equals to a 24 hour period when the default 5 minute polling period is used) may be retrieved for individual sites. The ESM site is set up as a standard web application and can be configured to use http or https protocol. In our example we have set up to use https so ESM site uses SSL encryption to protect the user authentication. Figure 1: ESM Initial Page And Login Screen. 3

7 Figure 2: Sample Data Center Summary Screen In ESM v.2.x Monitored sites can be divided amongst Data Centers which would then drill down to Servers and then sites on that server. In the summary screen above, one icon for each Data Center location is displayed. Here the status is normal. When there is an alert the icons would be flashing. Figure 3: Sample Hong Kong Data Center Summary Screen The above screen is the Data Center Summary Screen which displays status of all servers in one of the listed data centers. Here the status is normal. When there is an alert the icons would be flashing. 4

8 Figure 4: Sample EKP Sites on Server Summary Screen Status OK Any data center or server or site experiencing a problem will cause the icon to begin flashing yellow or red. Figure 5: Sample Data Center Alert Status Summary Screen Drilling down the Data Centers to the Hong Kong Data Center shows that NETD13 has triggered an alert. 5

9 Figure6A: Sample Server NETD13 In Hong Kong Data Center Alert Figure6B: Alternative View Showing All Servers in All Data Centers. Server NETD13 In Hong Kong Data Center Has Triggered Alert Clicking on NETD13 of the Hong Kong Data Center shows that alert was triggered by site exam.netdimensions.com 6

10 Figure 7: Site exam.netdimensions.com in NETD13 in Hong Kong Triggered Alert. Now that site has triggered an alert, the ESM site status will change and any 3 rd party application can be used to poll site status at regular intervals and be configured to send an SMS or pager alert when status is not OK. The url of the ESM Site Status to check is: - Page shows OK if all data center, servers, sites on servers are ok - Page shows a HTTP Status 503 The requested service is not available. Fig. 8: Once engineers have been notified by sms or pager that the ESM Site has triggered an alert they can log into the ESM site to identify which data center, server and site has triggered the alert & cause of alert. Now that we know alert was triggered by site Exam.netdimensions.com in NETD13 server in the Hong Kong Data Center, we can suspend the monitoring of that site while we investigate further. Click on Disable. 7

11 Figure 9: Site exam.netdimensions.com in NETD13 has Monitoring Stopped. In this example the error reported is Unable to Connect which indicates tomcat has failed for some reason. Simple resolution is to restart the tomcat service. Further steps would be to examine the tomcat & ekp logs to investigate why service stopped in first place. Now that exam.netdimensions.com site issue has been fixed we can resume monitoring of the site by clicking on Enable. Clicking on the site history will show for the past few days. The status of the polls will be stored to the database. Fig. 10: Site exam.netdimensions.com in NETD13 has error fixed and site restarted & monitoring resumed. Clicking on a site s Uptime will show the Uptime percent by month. All data is stored in database so can be retrieved later in sql queries. 8

12 Fig. 11: Site exam.netdimensions.com in NETD13 Uptime History Fig. 12: Last Error Report Summary For All Servers In All Data Centers 9

13 Installing ESM 2.x To install ESM, unzip the distribution zip file into the Tomcat webapps folder. You should install it on a machine independent of the sites being monitored, but this is not a requirement. - Unzip on to a Tomcat Instance and setup tomcat & apache virtual hosts - Create the virtual hosts for the monitor sites in apache httpd.conf & tomcat mod_jk.conf if setting up new one. ESM 2.x stores data onto an MSSQL database so create a db with its own user account & password. In this example we will create an ESM site called monitorus2 so the db will be called the same. Unzip the ESM package Unzip the ESM package and place inside a tomcat instance\webapps\monitorus2 folder or what ever name you want to call the ESM application. Create a MSSQL DB and run the schema\mssql\mssql.sql - Create a MSSQL DB with user name & password - run the <ESM_PACKAGE>\schema\mssql\mssql.sql - there may be an additional query for updates. o See Appendices for full mssql.sql Edit the <TOMCAT_HOME>\webapp\monitorus2\WEB-INF\conf\sitelist.properties - This sitelist.properties file is very similar to an ekp.properties file o A Sample sitelist.properties with Key Parameters is listed below: default.user=monitorus2 default.password=<xxxxxxx> # Microsoft SQL Server 2005 (Microsoft JDBC Driver for SQL Server 2005) default.drivername=com.microsoft.sqlserver.jdbc.sqlserverdriver default.connecturl=jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databasename=monitorus1;s electmethod=cursor default.dbtype=mssql # # Authorized users List each system.#_userid=<unername> and password system.1_userid=ndadmin system.1_password=<xxxxxxxxx> # # Application domain identifier system.domain=monitorus2 ## these 2 parameters are in ESM1.x but not in 2.0 so have placed here # Number of intervals for computing average poll response time (aim for 30 minutes total) system.responseaverageinterval=6 # Security violation threshold per 1000 page hits system.securityviolationrate=50 10

14 ##### site to be monitored format set to None as it is a non-ekp web site site10= site10.datacenter=canada site10.server= site to be monitored site10.format=none site10.target= # NETD13 Server site12=exam.netdimensions.com site12.datacenter=hong Kong site12.server=netd13 site12.format=statusformat1 site12.target= ## repeat for each site to be monitored o See Appendices for full sitelist.properties Any time a new site is added or other change made you can restart the tomcat containing ESM or click on the ESM Admin button & then click Reinitialize Site List to read in new changes made to the sitelist.properties. The contents of the sitelist file may be changed and re-read by the ESM without needing to restart the service. There are currently two types of sites that may be configured: No Status Detail Available: Any (even non-ekp) site may be configured with a type of "site.format=none", which instructs the ESM to merely check that the target returns a response. STATUSFORMAT1 Information Returned: A site configured as "site.format=statusformat1" is assumed to return a message containing detailed EKP status attributes. To use this message type, you must have EKP , EKP or EKP , or later. 11

15 Create the virtual host settings inside your web server In our example we use Apache with SSL httpd.conf ##### monitorus2.netdimensions.com ##### <VirtualHost xxx.xxx.xxx.xxx:80> ServerAdmin DocumentRoot "<TOMCAT_HOME>\webapps\monitorus2" FileETag None ServerName monitorus2.netdimensions.com RewriteEngine On RewriteRule ^/$ [R,L] RewriteRule ^/index\.html$ [R,L] FileETag None CheckSpelling On ErrorLog logs/monitorus2-error.log CustomLog logs/monitorus2-access.log common </VirtualHost> <Directory <TOMCAT_HOME>\webapps\monitorus2> FileETag None Options FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory> Apache SSL.CONF #<VirtualHost monitorus2.netdimensions.com:443> <VirtualHost :443> ServerAdmin DocumentRoot "<TOMCAT_HOME>\webapps\monitorus2" ServerName monitorus2.netdimensions.com ErrorLog logs/monitorus2-error.log CustomLog logs/monitorus2-access.log common TransferLog logs/monitorus2-access.log SSLEngine on ReWriteEngine On. SSL config. </VirtualHost> Create the virtual host settings inside your Tomcat Server Edit <TOMCAT_HOME>\conf\jk\mod_jk_multiple_instances.conf # # # # Virtual Host:/monitorus2 # # # # # Static files Alias /monitorus2 "d:/nd/tomcat5_inst2/webapps/monitorus2" <Directory "d:/nd/tomcat5_inst2/webapps/monitorus2"> Options -Indexes FollowSymLinks 12

16 DirectoryIndex index.html index.htm index.jsp </Directory> # Deny direct access to WEB-INF and META-INF # <Location "/monitorus2/web-inf/*"> AllowOverride None deny from all </Location> <Location "/monitorus2/meta-inf/*"> AllowOverride None deny from all </Location> # # Use Directory too. On Windows, Location doesn't work unless case matches # <Directory "d:/nd/tomcat5_inst2/webapps/monitorus2/web-inf/"> AllowOverride None deny from all </Directory> <Directory "d:/nd/tomcat5_inst2/webapps/monitorus2/meta-inf/"> AllowOverride None deny from all </Directory> JkMount /monitorus2/servlet/* tomcat_2 Start up tomcat & check for errors <TOMCAT_HOME>\webapps\monitorus2\WEB-INF\logs\esm.log Typical errors may be due to incorrect db, username or password specified in the sitelist.properties. If everything is correctly set up you will see the ESM login page at the url you configured. e.g. The status monitor display may then be accessed via a browser by pointing it to <your site>/ekpstatusmonitor/servlet/reportstatus (an initial screen will prompt for a password to ensure that outside users do not have access to your site information). The ESM site is set up as a standard web application and can be configured to use http or https protocol. In our example we have set up to use https so ESM site uses SSL encryption to protect the user authentication. 13

17 The EKP Status Monitor (ESM) is a program that periodically polls a set of EKP sites to verify that they are functioning correctly. Current activity statistics are returned to a browser client which will then display the results using either a server summary screen or a site-bysite detail information screen. In addition, activity data for the previous "n" polling periods (a configurable value that defaults to a days worth of data)) may be retrieved for individual sites. Retrieved performance data for each site may be persisted to a configured database, although if a database is not available ESM will continue to monitor as it did in previous versions. 14

18 Requirements The ESM runs as a servlet application, hence any servlet container (e.g. Tomcat) with JSDK 1.5 or later is suitable. While ESM may poll any type of site, it intelligently interprets information from EKP, and is most useful with sites that have been upgraded to the following minimum levels. EKP EKP EKP EKP 4.5.x.x (all) EKP 4.6.x.x (all) These versions have a special transaction (STATUSFORMAT1) that returns detailed statistical data to the ESM. Earlier versions can be monitored, but they can only be tested for an up/down status, as well as polling response time. Additionally, EKP and later return even more operational information using the same format transaction. 15

19 Configuration Each site is defined to the ESM in the.../conf/sitelist.properties file. The format for the configuration statements are documented in the properties file itself, but a sample site definition would look similar to the following: site21=gold.netdimensions.com site21.server=netd2 site21.format=statusformat1 site21.target= The contents of the sitelist file may be changed and re-read by the ESM without needing to restart the service. There are currently two types of sites that may be configured: No Status Detail Available: Any (even non-ekp) site may be configured with a type of "site.format=none", which instructs the ESM to merely check that the target returns a response. STATUSFORMAT1 Information Returned: A site configured as "site.format=statusformat1" is assumed to return a message containing detailed EKP status attributes. To use this message type, you must have EKP , EKP or EKP , or later. 16

20 STATUSFORMAT1 Data Reported Data for STATUSFORMAT1 sites are the most useful as this includes the following operational information: Bootup Time License Expiration Time Response Time EKP Release Version #Currently Active Connections #Currently Allocated Connections #Max Connections #Currently Active Sessions #Max Active Connections #Total Logins #Total System Errors #Total TX Hits # s Sent #Security Violations #Enrollments #Module Launches #Uploads #Reports #OPML Imports #OPML Exports In addition to returning the above information, a STATUSFORMAT1 request does some internal integrity checking within each EKP instance. For example, EKP verifies that connections can still be allocated and the database successfully contacted. 17

21 Alert Criteria After processing the above statistical data, ESM will evaluate threshold settings for current activity (which includes some derived figures that are interval specific). Each site is then marked as still functioning normally (a Green status) or in a Yellow Warning or Red Alert state. The following conditions are noted as a part of the analysis: ALERT: The site cannot be contacted; ALERT: The status code returned from the site indicates an internal error; ALERT: All connections are in use; ALERT: The response time exceeds the alert threshold (e.g. 45 sec.); WARNING: All connections are allocated; WARNING: 80% of connections are in use; WARNING: The response time exceeds the warning threshold (e.g. 15 sec.); WARNING: The system error rate exceeds the allowed interval amount (e.g. 10 in 5 minutes); WARNING: The queue exceeds a specified threshold (e.g. default is 2000); WARNING: The transport encounters connection errors to the SMTP server; WARNING: The average STATUSFORMAT1 poll response time for the past 30 minutes is unusually high; WARNING: The security violation rate (e.g. 25 per 1000 TX requests) is suspicious, and needs investigation; The icon-oriented server summary screen will cause a server icon to flash in the appropriate color if ANY site on that server has encountered a problem. The threshold settings for the above conditions are set using configuration parameters in the "sitelist.properties" file that is read upon startup of the ESM. The security violation alert in EKP is mainly related to the login and/or password being incorrect. EKP can tighten the login attempt and password complexity. E.g. after 5 times of login attempts, it should be suspended and until further action from administrator. You can check the failed sessions/attempts in Manage -> system admin Mgr ->Access Violation report Which will list the date/time IP address and login id that the failed attempts were made from, Other causes of Security Violations are : The basic check is whether or not a user attempting to execute a specific transaction has permissions. Problems are most likely because: 1. The role is configured incorrectly, so users have an menu item showing whereas access to some underlying function is turned off; 18

22 2. Users are hacking EKP by trying to type in an URL with some TX that they do not have permissions to in the role. 3. Most likely, users have bookmarked a specific page (maybe their training record or home page) that they sometimes click on while not logged in. Login failures are a separate count and are not part of this particular total. This is why EKP or the EKP Status Monitor (ESM) may report a security violations when there s only a handful of users on site or when there are no or few failed login attempts recorded. ESM Parameter listed in ESM Help WARNING: The security violation rate (e.g. 25 per 1000 TX requests) is suspicious, and needs investigation; # Security violation threshold per 1000 page hits system.securityviolationrate=50 Other ways to protect EKP content resources are to use the Secure Servlet Filters to protect folders that contain sensitive content. Such as the courseware folders in nd/fresco content, courses, dataloader, batchreports, filerepository, repository, uploads folders. With SecureServlet Filter enabled for listed folders, access to those folders will prompt user to login - thus protecting content in folders from external users who have not authenticated. See EN200_SecureFilter_Setup_Guide_V1_1.pdf on the support site or wiki or from the ekp online help or cdroms. 19

23 Red Alert Interface A special URL (.../monitorus2/servlet/redalertsummary) may be polled by an external source, such as the Red Alert site monitoring service. If any site has an error condition, a 503 HTTP error (service not available) is returned. Otherwise, a normal OK is returned. This capability is useful for services such as Red Alert that provide phone or SMS callout for site support conditions. Individual sites may be configured to be monitored but data not archived ("site.disabledataarchiving=true") or configured to be monitored but NOT trigger an external alert (("site.disableexternalalert=true") - both are useful for test and demo sites that are a part of the ESM environment. Additionally, the "system.externalalertsensitivity=n" configuration parameter may be used to control the overall sensitivity of reporting data center level errors. Here, the 503 error will only be reported if an error occurs "n" times in a row for a specific site. This helps to cut down on reporting of errors for transient (e.g. it only happens once) conditions. NetDimensions uses a service called Red Alert that is able to poll web sites and to issue SMS messages to designated support personnel if a problem is encountered. ESM makes available a special URL that may be polled by such a service, consolidating the status of all monitored sites into a single check point. NOTE - Any 3 rd party Web Site Monitoring Application can be used to monitor the ESM site and be set to send out alerts via SMS or Pager messages. Examples are KeyNote s Red Alert (redalert.com) service or Alertra ( The url of the ESM Site Status to check is: - Page shows OK if all data center, servers, sites on servers are ok - Page shows a HTTP Status 503 The requested service is not available. For example, if 20 sites are being monitored but one is in alert status, the ESM will report an error to Red Alert when it is polled, thus resulting in a callout the support manager. It is much more advantageous for the ESM to be polled, since it performs a more thorough health check than simple polling services are able to do on their own. Important Note You must make additional arrangements for the callout support service. You may contact for more information about one such service (used by NetDimensions), although there are a number of others available on the Internet. When using such a service, note that only one system (the ESM) needs to be monitored by the service, thus minimizing the overall cost for automatic telephone callouts. 20

24 Appendix A: Sample sitelist.properties File Sample <TOMCAT_HOME>\webapps\monitorus2\WEB-INF\confsitelist.properties ################################################################################## ### # EKP STATUS MONITOR PROPERTIES # Each site is defined with the following attributes: # # site<n> - An arbitrary name; # site<n>.server - the server containing this site. # site<n>.format - the format of a status response (NONE or STATUSFORMAT1) # site<n>.target - the URL used for the status poll # site<n>.priority - HIGH/LOW. LOW implies two error cycles are required for # RedAlert reporting # The default is "HIGH". # site<n>.disabled - true/false, indicates that the site is not polled. # The default is "false". # # Sites on the same server should be adjacent to each other in this file to facilitate # reporting, since the overall list is not currently sorted when read in from this file. # ################################################################################## ### # # The "default" connection pool is used to access the ESM database. default.name=default default.initialconnections=2 default.maxconnections=10 default.incrementconnections=2 default.retries=5 default.interval=1000 default.user=<username-xxx> default.password=<password-xxx> # This setting enables the ESM to be resilient in the event the database server # needs to restart, as it can automatically re-connect once the DB is back on-line default.connectionpooltestinterval=1 default.connectionpoolreplacementinterval=400 # Microsoft SQL Server 2005 (Microsoft JDBC Driver for SQL Server 2005) default.drivername=com.microsoft.sqlserver.jdbc.sqlserverdriver default.connecturl=jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databasename=monitorus2;s electmethod=cursor default.dbtype=mssql #--- Username & passwords to log into ESM Site # Authorized users system.1_userid=ndadmin system.1_password=xxxxxx system.2_userid=userxxx system.2_password=passwordxxx # system.debug=false 21

25 system.nthreads=0 #External RedAlert sensitivity level. # indicates the system.externalalertsensitivity=3 # The time (in ms.) between polling intervals. system.refreshtimems= # The # sample intervals to save as history. system.maxhistoryentries=1500 # The threshold at which to issue a warning for slow response. system.warningresponsetimems=23000 # The threshold at which to issue an alert for slow response. system.alertresponsetimems=45000 # The # of errors allowed in one sample interval.. system.systemerrorrate=20 # Application domain identifier system.domain=monitorus2 system.responseaveragethresholdms=15000 ## these 2 parameters are in ESM1.x but not in 2.0 so have placed here # Number of intervals for computing average poll response time (aim for 30 minutes total) system.responseaverageinterval=6 # Security violation threshold per 1000 page hits system.securityviolationrate=200 ### STATUSFORMAT1 is used for EKP Sites 4.0.x and up ### Other non-ekp sites should use format=none ### NETD4 - site10=pwcasia.netdimensions.com site10.datacenter=hong Kong site10.server=netd4 site10.format=statusformat1 site10.target= site10.priority=low ### NETD12 site20=netd12.netdimensions.com:10080/studio site20.datacenter=hong Kong site20.server=netd12 site20.format=none site20.target= #site20.priority=low site21=ecp1.netdimensions.com/studio site21.datacenter=hong Kong site21.server=netd12 site21.format=none site21.target= #site21.priority=low 22

26 site22=ecp2.netdimensions.com/studio site22.datacenter=hong Kong site22.server=netd12 site22.format=none site22.target= #site22.priority=low 23

27 ESM v2.01 MSSQL.SQL SCHEMA create table esm_history ( site_name nvarchar(100) not null, date_polled datetime not null, bootup_date nvarchar(50) not null, status_code nvarchar(50) not null, status_msg nvarchar(100) not null, ekp_version varchar(50) not null, n_active_connections int not null, n_alloc_connections int not null, n_max_connections int not null, n_active_sessions int not null, n_max_sessions int not null, n_total_logins int not null, n_sys_errors int not null, n_page_hits int not null, n_ext_ s int not null, _queue_size int not null, n_ _trans_errs int not null, n_security_errors int not null, n_enrollments int not null, n_uploads int not null, n_reports int not null, n_launches int not null, n_tests int not null, tx_response_ms int not null, page_response_ms int not null, n_opml_imports int not null, n_opml_exports int not null, interval_sys_errs int not null, interval_page_hits int not null, interval_ s int not null, interval_security_errs int not null, interval_enrollments int not null, interval_uploads int not null, interval_reports int not null, interval_launches int not null, interval_sessions int not null, interval_tests int not null, interval_opml_imports int not null, interval_opml_exports int not null, interval_tx_rate float not null, site_restarted char(1) not null, constraint PK_ESM_HISTORY primary key (site_name, date_polled) ) go create table esm_heartbeat ( appname nvarchar(100) not null, version nvarchar(20) not null, heartbeat_date timestamp null, constraint PK_ESM_HEARTBEAT primary key (appname) ) go insert into esm_heartbeat (appname, version) values ('esm', '2.0') go 24