Enterprise Knowledge Platform 5.6

Transcription

1 Enterprise Knowledge Platform 5.6 EKP Multiple Instance Configurations for Apache2/Tomcat4

2 Document Information Document ID: EN144 Document title: EKP Multiple Instance Configurations for Apache/Tomcat Version: 1.5 Document date: 15 April 2009 This document may be revised from time to time. Please check NetDimensions Support site at for updates to this and other documents or send an to to request the most recent version. Please report any errors or feedback with this document by sending an to Copyright Information Copyright by NetDimensions Ltd. All Rights Reserved. Information in this document is subject to change without notice. The software described herein is furnished under a license agreement, and it may be copied only in accordance with the terms of that agreement. No part of this publication may be reproduced, transmitted, or translated in any form or by any means without the prior written permission of NetDimensions Ltd. All company and product names used herein may be trademarks or registered trademarks of their respective companies unless stated otherwise. How to Contact NetDimensions Support US toll-free number support@netdimensions.com General Enquiries info@netdimensions.com

3 Table of Contents Description...1 About This Guide... 1 If You Need Help... 1 Preparing Individual EKP Instances...2 Intended Use... 2 Assumptions... 2 Setting up Additional EKP Databases... 2 Setting up Custom Login Pages... 3 Shared JVM Setup Steps...4 Overview... 4 Setting up Additional EKP Instances... 5 Setting up Tomcat... 7 Setting up Apache... 8 Optimizations... 9 Troubleshooting Private JVM Setup Steps Overview Setting up Additional Tomcat Instances Setting up Additional EKP Instances Setting up Tomcat Setting up the worker.properties Setting up Apache Optimizations Troubleshooting Load Balancing Configurations Overview How Does This Configuration Work? Basic Requirements STEP 1 -- Shared Web Site Folder STEP 2 -- Setting up Apache STEP 3 -- Setting up Tomcat Sample Tomcat server.xml Configuration File... 31

4 Description About This Guide The purpose of this paper is to describe the basic configurations steps needed to setup multiple independent instances of EKP on the same server machine. This capability is particularly useful when setting up EKP for testing, development, or hosting operations where you want to run separate instances on the same system in order to leverage economies of scale with existing system resources. Although EKP can be setup in any J2EE environment to run in this manner, this document describes the setup for Apache2 and Tomcat 4.1, and applies to all versions of EKP. The principles are the same for other server software components. Two basic techniques are covered: Setting up multiple EKPs sharing a single Tomcat and single JVM. This technique is straightforward, uses fewer system resources, and is best for smaller machines and a smaller # instances. Setting up multiple Tomcats, each with one or more EKP instances and each with a private JVM. This technique allows each instance to be brought up/down independently of the others, and is more suitable for large scale SMP systems where management flexibility and scalability are most important. The description of the individual steps in this document is intended to cover the basic requirements in each area and is not a substitute for reading associated documentation for each product. If You Need Help If you cannot resolve a configuration problem using this guide or the online help, or if you should have any queries related to the technology employed within EKP, your first line of contact should be as described in your Technical Support Contract. For other queries, or if you are not sure whom to contact, NetDimensions Ltd. may be contacted at info@netdimensions.com. Please also refer to the support section of the NetDimensions web site at for the latest information regarding various services. 1

5 Preparing Individual EKP Instances Intended Use It is assumed in this document that you are preparing multiple EKPs because each is intended to function independently of the others. For example, setting up EKP for different development teams or different clients in a hosting service. So, for each EKP instance, you will want to do the following before you even begin to configure Apache and Tomcat: Set up a new EKP database for each new instance; Customize the login page and skins so that each site is suitable for the intended end user. This document does not discuss setting up a cluster of EKP instances to access a shared database. Assumptions You should already have installed a standard EKP with Apache2 and Tomcat 4.1 from the installation CD, as the creation of additional instances is greatly simplified when you are performing this setup for an existing single EKP environment. So, this document does not cover the setup steps for the standard EKP installation as is typically done from the release CD refer to the EKP Installation Guide for specific instructions regarding general setup. The focus in this document is on configuration changes to support multiple instances, not individual EKP setup steps. Note that the contents of the individual EKP site directories may ultimately be different if you are running a server with different versions of EKP (i.e. EKP 3.0 and EKP 4.0), but the overall configuration within Apache and Tomcat is unaffected by these individual site differences. Setting up Additional EKP Databases You will want to setup a separate database for each new instance of EKP. The database utilities involved in doing this are database specific, but the steps are identical across databases. Microsoft SQL Server is used in the example below. 1. Using SQL Server Enterprise Manager, Create a new database for each instance of EKP. For example, ekp_db_1, ekp_db_2, ekp_db_3, etc. The choice of the new database name is arbitrary. 2. You may want to setup a unique login for each database (instead of the default ndadmin, but this is more of an operational decision based on who will be allowed to manage each instance. 3. Each database should be a minimum of 50MB, and as large as your intended use dictates 50MB is suitable for general testing purposes, whereas as 1GB is more 2

6 suitable for a production site that must serve users. Other options can be setup according to your operational requirements as EKP does not itself have specific constraints regarding the database management settings. 4. Use the appropriate EKP schema definitions file from the EKP release CD to create the user tables in each database. For example, this can be found in the <install directory>setup/schema/ directory on your hard disk. Typical utilities suitable for processing the EKP schema file are: a. Microsoft SQL Query b. Sybase isql or SQL Advantage; c. Oracle -- sqlplus For example, isql Undadmin_1 P < sybase.sql (where this userid defaults to the new database). Setting up Custom Login Pages The default EKP login page is: <tomcat>webapps/ekp/index.html For each new instance, you will probably want to customize the appearance of the login page. This step is optional (though highly recommended to avoid confusion for users of multiple sites). At a minimum, you must change the URL that submits the userid/password so that it references the correct name for each instance. For example, in index.html you might change: <form method="post" action="/ekp/servlet/ekp" target="_top"> to <form method="post" action="/ekp_1/servlet/ekp" target="_top"> if ekp_1 is the name of one of the new EKP instances in webapps. 3

7 Shared JVM Setup Steps Overview This method allows you to create additional EKP instances under the current Tomcat server by invoking them using different application names in the webapps directory. EKP automatically recognizes the application name it is running under, so most of the configuration steps are needed to identify each application name to Apache and Tomcat. The diagram below illustrates the basics of this configuration. EKP1 Apache Tomcat EKP1 EKP3 Diagram 1: Basic Configuration 4

8 Setting up Additional EKP Instances For each new instance of EKP: 1. Go the currently installed Tomcat directory <tomcat>\webapps\. 2. Copy the current ekp application as ekp_1, ekp_2,, ekp_n (or any arbitrary name you need to fit your environment) for as many sites as you want to create (see Figure 1 as an example). This will duplicate the entire web site and application run time environment for each instance. 3. Modify the \webapps\ekp_n\web-inf\conf\ekp.properties file for each new instance, setting the values of the system.baseurl and system.domain properties appropriately. system.baseurl= system.domain=ekp_n 4. Modify the \webapps\ekp_n\web-inf\conf\ekp.properties file for each new instance to reference the new associated database. # # Microsoft SQL Server (Microsoft JDBC Driver) default.drivername=com.microsoft.jdbc.sqlserver.sqlserverdriver default.connecturl=jdbc:microsoft:sqlserver://abc:1433;databasename=ekp_db_n;select Method=cursor default.dbtype=mssql 5. Modify the..\webapps\ekp_n\index.html login page to refer to the current instance. For example, change : <form method="post" action="/ekp/servlet/ekp" target="_top"> to <form method="post" action="/ekp_n/servlet/ekp" target="_top"> You may also want to change the displayable contents of the index.html login page to uniquely identify each site, but that is a matter of how you want to use the new instance, not a technical requirement. 5

9 Figure 1: Tomcat Webapps Directory 6

10 Setting up Tomcat The file mod_jk.conf defines the mappings to be used for each application defined to Apache and Tomcat. Each new EKP instance should have the following configuration statements added to the <tomcat>\conf\jk\mod_jk.conf file (replace the ekp_n name with your individual instance name): #################### Application /ekp_n #################### # Static files Alias /ekp_n "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n" <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n"> Options -Indexes FollowSymLinks DirectoryIndex index.html index.htm index.jsp </Directory> Alias /nd "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/nd" <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/nd"> Options -Indexes FollowSymLinks DirectoryIndex index.html index.htm index.jsp </Directory> # Deny direct access to WEB-INF and META-INF # <Location "/ekp_n/web-inf/*"> AllowOverride None deny from all </Location> <Location "/ekp_n/meta-inf/*"> AllowOverride None deny from all </Location> # # Use Directory too. On Windows, Location doesn't work unless case matches # <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/WEB-INF/"> AllowOverride None 7

11 </Directory> deny from all <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/META-INF/"> AllowOverride None deny from all </Directory> JkMount /ekp_n/tx/* ajp13 JkMount /ekp_n/servlet/* ajp13 JkMount /ekp_n/*.jsp ajp13 JkMount /ekp_n/servlet/ekp/* ajp13 JkMount /ekp_n/*.tx ajp13 Note This example assumes that your worker connection, as defined in <tomcat>\conf\jk\worker.properties has the name ajp13 as shown below. # BEGIN workers.properties # Definition for Ajp13 worker worker.list=ajp13 worker.ajp13.port=8009 worker.ajp13.host=localhost worker.ajp13.type=ajp13 # END workers.properties Setting up Apache For each new instance of EKP, you can (optionally) setup a specific virtual host name by adding an entry such as the following to the <apache2>\config\httpd.conf file (replace the ekp_n instance name with your individual instance name, and other highlighted items as appropriate for your site): ##### ekp_n.netdimensions.com ##### <VirtualHost :80> ServerAdmin support@netdimensions.com DocumentRoot "d:\nd\tomcat4.1.27\webapps\ekp_n" ServerName ekp_n.netdimensions.com 8

12 ErrorLog logs/ekp_n-error.log CustomLog logs/ekp_n-access.log common </VirtualHost> <Directory d:\nd\tomcat4.1.27\webapps\ekp_n> Options FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory> This would allow you to reference a particular site using an URL of the following type: However, if you do not need to setup separate virtual hosts for each ekp instance, as you merely want to access each application independently from a current server domain, you could ignore the above virtual host setup and access each application as follows: For this latter approach to work, you may need to add a context statement for each application in the <Tomcat>\conf\server.xml file. For example: <Context path="/ekp_n" docbase="c:\tomcat\webapps\ekp_n" reloadable="false" /> 9

13 Optimizations Running multiple EKP instances in a single Java Virtual Machine with a single Tomcat places greater strain on the JVM. Therefore, you need to ensure that your JVM is appropriately configured with adequate memory. If you are running Tomcat as a service, your registry entries should appear similar to the following: Figure 2: Registry Entries Key points from the above setting are: At least 128MB is allocated to the JVM, up to 256MB. This is suitable for a few EKP instances (depending upon activity and loading). More is also OK. Note the setting of the JVM Option Count parameter! JVM 1.3.1_08 is used, server version! If you use earlier versions of the JVM or the hotspot JVM, you can exhaust memory with only 4-6 EKP instances and have the JVM terminate abnormally. If you do not add more memory to the JVM (e.g. you use the defaults, you will encounter JVM out of memory conditions after a short period of use with multiple instances. 10

14 EKP LIbraries In the <tomcat>\webapps\ekp_n\web-inf\lib directory you will find a number of libraries used in support of EKP. You must ALWAYS put ekp.jar and chatserver.jar here. Optionally, other jars in WEB-INF\lib can be moved to <tomcat>\shared\lib as a way to save some JVM executable memory. DO NOT PUT ekp.jar or chatserver.jar in the Tomcat shared library folder. This should only be done if all of the EKP sites are the same Release level (e.g. EKP 3.2.2) as the libraries, otherwise keep them all in the WEB-INF\lib folder. Troubleshooting If you are having problems, make sure each component is working properly in a step-by-step manner, thus eliminating possible sources of the problem. 1. You can verify the database is up by logging into it directly (using SQL query tools) and connecting to the various databases to ensure they were created correctly). 2. You can verify that Tomcat is up by entering an URL such as (this only works if you have configured Tomcat to have an HTTP listener on 8080): If Tomcat is setup to listen on port 8080; you will get a generic Tomcat response. This does not yet test EKP communications. 3. You can check each EKP instance mapping under Tomcat by entering an URL such as: This only gets you the login page but does not yet start EKP. If you do not get this far, your web.xml file is incorrect. Go ahead and login to see if the EKP instance is running and connected to the database. Change the login defaults for each one to ensure you are not accidentally getting the wrong instance. 4. Now, under Apache, enter an URL such as: If you cannot get this far, your mod_jk.conf file is probably incorrect. 5. If you are using virtual hosts, test a virtual host name by entering: 11

15 If this does not result in the appropriate instance login page, likely your Apache httpd.conf file is incorrect. 12

16 Private JVM Setup Steps Overview This method allows you to setup each EKP instance with its own JVM. The principal advantages of this approach are: You can bring each instance up/down independently of the others; The Individual Tomcats can actually be running on other machines; Greater resiliency in case one JVM crashes (e.g. out-of-memory, etc.) More effective utilization of large SMP machines The diagram below illustrates the basics of this configuration. Tomcat EKP1 Apache Tomcat EKP2 Tomcat EKP3 Diagram 2: Basics of Configuration 13

17 Setting up Additional Tomcat Instances For each new instance of Tomcat 1. Copy the existing Tomcat instance as multiple directories as shown in the figure. 2. Create multiple Tomcat services as follows: a. In regedit, export the existing Tomcat service instance. b. Using notepad, rename the directories to correspond to the new instance. c. In regedit, import the new instance registry file. d. Repeat as many times as there are instances to add. e. Reboot the machine so that the new services are available. It is a good idea to make several additional disabled services that can be used later without all of these setup steps. Setting up Additional EKP Instances For each new instance of EKP: 3. Go the currently installed Tomcat directory <tomcat>\webapps\. 4. Copy the current ekp application as ekp_1, ekp_2,, ekp_n for as many sites as you want to create (see Figure 2 as an example). This will duplicate the entire web site and application run time environment for each instance. 5. Modify the \webapps\ekp_n\web-inf\conf\ekp.properties file for each new instance, setting the values of the system.baseurl and system.domain properties appropriately. system.baseurl= system.domain=ekp_nmodify the \webapps\ekp_n\web-inf\conf\ekp.properties file to reference the new associated database. # # Microsoft SQL Server (Microsoft JDBC Driver) default.drivername=com.microsoft.jdbc.sqlserver.sqlserverdriver default.connecturl=jdbc:microsoft:sqlserver://abc:1433;databasename= ekp_db_n;selectmethod=cursor default.dbtype=mssql 14

18 6. Modify the..\webapps\ekp_n\index.html login page to refer to the current instance. For example, change : <form method="post" action="/ekp/servlet/ekp" target="_top"> to <form method="post" action="/ekp_n/servlet/ekp" target="_top"> You may also want to change the displayable contents of the index.html login page to identify the different site, but that is a matter of how you want to use the new instance, not a technical requirement. Figure 3: Tomcat Webapps Directory 15

19 Setting up Tomcat The file mod_jk.conf defines the mappings to be used for each application defined to Apache and Tomcat. Each new instance should have the following configuration statements added to the <tomcat>\conf\jk\mod_jk.conf file: Note that each ekp instance is mapped to a specific ajp13 worker which is arbitrarily named tomcat_n. Note Although you need to describe each EKP and the list of workers needed to communicate with Tomcat, the mod_jk.conf file is actually read by Apache even though it is by convention placed in the <tomcat>\conf directory. So, you can pick one of the Tomcat instances to hold the <jk> configuration files there is no need to duplicate it for each Tomcat. #################### Application /ekp_n #################### # Static files Alias /ekp_n "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n" <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n"> Options -Indexes FollowSymLinks DirectoryIndex index.html index.htm index.jsp </Directory> Alias /nd "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/nd" <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/nd"> Options -Indexes FollowSymLinks DirectoryIndex index.html index.htm index.jsp </Directory> # Deny direct access to WEB-INF and META-INF # <Location "/ekp_n/web-inf/*"> AllowOverride None deny from all </Location> <Location "/ekp_n/meta-inf/*"> AllowOverride None 16

20 deny from all </Location> # # Use Directory too. On Windows, Location doesn't work unless case matches # <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/WEB-INF/"> AllowOverride None deny from all </Directory> <Directory "C:/Program Files/Apache Group/Tomcat 4.1/webapps/ekp_n/META-INF/"> AllowOverride None deny from all </Directory> JkMount /ekp_n/tx/* tomcat n JkMount /ekp_n/servlet/* tomcat n JkMount /ekp_n/*.jsp tomcat n JkMount /ekp_n/servlet/ekp/* tomcat n JkMount /ekp_n/*.tx tomcat n Setting up the worker.properties The worker.properties file is used to define connections/channels between Apache and Tomcat. Since we are going to be running multiple Tomcats, we need to define multiple or ajp13 workers. A sample worker.properties file would look like this: # BEGIN workers.properties worker.list=tomcat_1, tomcat_2, tomcat_3, tomcat4, tomcat_5 # Definition for Ajp13 worker for Tomcat Instance #1 worker.tomcat_1.port=8109 worker.tomcat_1.host=localhost worker.tomcat_1.type=ajp13 # Definition for Ajp13 worker for Tomcat Instance #2 worker.tomcat_2.port=8209 worker.tomcat_2.host=localhost worker.tomcat_2.type=ajp13 # Definition for Ajp13 worker for Tomcat Instance #3 17

21 worker.tomcat_3.port=8309 worker.tomcat_3.host=localhost worker.tomcat_3.type=ajp13 # Definition for Ajp13 worker for Tomcat Instance #4 worker.tomcat_4.port=8409 worker.tomcat_4.host=localhost worker.tomcat_4.type=ajp13 # Definition for Ajp13 worker for Tomcat Instance #5 worker.tomcat_5.port=8509 worker.tomcat_5.host=localhost worker.tomcat_5.type=ajp13 # END workers.properties Setting up Apache For each instance of EKP, you can (optionally) setup one or more specific virtual host names by adding an entry such as the following to the <apache2>\config\httpd.conf file: ##### ekp_n.netdimensions.com ##### <VirtualHost :80> ServerAdmin support@netdimensions.com DocumentRoot "d:\nd\tomcat4.1.27\webapps\ekp_n" ServerName ekp_n.netdimensions.com ErrorLog logs/ekp_n-error.log CustomLog logs/ekp_n-access.log common </VirtualHost> <Directory d:\nd\tomcat4.1.27\webapps\ekp_n> Options FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory> This would allow you to reference a particular site using an URL of the following type: 18

22 However, if you do not need to setup separate virtual hosts for each ekp instance, as you merely want to access each application independently from a current server domain, you could ignore the above virtual host setup and access each application as follows: Optimizations Tomcat services are setup exactly as the service for the shared JVM service it s just that you have multiple services to configure (and hence their file paths below would each be different). Figure 4: File Paths 19

23 Key points from the above setting are: EKP Libraries Since (probably) a single EKP is running in each Tomcat you may want to set the minimum memory lower (e.g. Xms64m) for each service. If you have memory to burn, more certainly will not hurt, but is probably not necessary for small sites. Make sure the JVM Option Count parameter is set correctly! Here, JVM 1.3.1_08 is used, server version. If you use earlier versions of the JVM or the hotspot JVM, you can more quickly exhaust memory and have the JVM terminate abnormally. In the <tomcat_n>\webapps\ekp_n\web-inf\lib directory you will find a number of libraries used in support of EKP. You must ALWAYS put ekp.jar and chatserver.jar here. Optionally, other jars in WEB-INF\lib can be moved to <tomcat_n>\shared\lib as a way to save some JVM executable memory. DO NOT PUT ekp.jar or chatserver.jar in the Tomcat shared library folder. Troubleshooting If you are having problems, make sure each component is working properly in a step-by-step manner, thus eliminating possible sources of the problem. 6. You can verify the database is up by logging into it directly (using SQL query tools) and connecting to the various databases to ensure they were created correctly). 7. You can verify that Tomcat is up by entering an URL such as : If Tomcat is setup to listen on port 8080; you will get a generic Tomcat response. This does not yet test EKP. 8. You can check each EKP instance mapping under Tomcat only by entering an URL such as: This only gets you the login page but does not yet start EKP. If you do not get this far, your web.xml file is incorrect. Go ahead and login to see if the EKP instance is running and connected to the database. Change the login defaults for each one to ensure you are not accidentally getting the wrong instance. 20

24 9. Now, under Apache, enter an URL such as: If you cannot get this far, your mod_jk.conf file is probably incorrect. If you are using virtual hosts, test a virtual host name by entering: If this does not result in the appropriate instance login page, likely your httpd.conf file is incorrect. 21

25 Load Balancing Configurations Overview For load balancing configurations, you will setup multiple semi-independent EKP instances, each with its own JVM and application server, but sharing the same database and web server. The principal advantages of this approach are: Pros Scalability regarding server CPU/memory/disk activity, with each instance running on a different machine and the ability to add additional nodes as the workload increases; Greater resiliency in case one server crashes (hardware failure, etc.), although not all configurations are necessarily more resilient. Cons More machines to manage; More complex setup and configuration; Greater costs; Does not address network scalability, which is typically the first bottleneck encountered in large systems. If you are setting up such a configuration, your first step should be to decide whether the priority is to be on scalability, resiliency, or both, as these objectives have a strong bearing on the configuration choices and cost. The diagram below illustrates the basics of this type of configuration when multiple EKP instances are setup to offer both load balancing and some resiliency. 22

26 MachineA MachineB MachineD Tomcat1 EKP1 Apache L B W Shared Database Tomcat2 EKP2 MachineC Diagram 3: Sample Load Balancing Configuration This sample configuration is but one of numerous examples of how systems can be arranged to offer scalability and resiliency, but it is not the only such combination of machines. Each of the above components could, technically, all be configured to run on a single machine, but that would be defeating the purpose of creating this type of configuration as it would eliminate the advantages this combination of components offers. Note The configuration details and capabilities of Load-Balancing and Resiliency solutions are also very much dependent upon the capabilities of the selected Application Server (e.g. JRUN, Resin, WebSphere, Tomcat, etc.). Most commercial products put in considerable additional engineering effort to offer features that make their product unique or especially valuable in this area. For the purposes of this paper, the EKP configuration handling is described in terms of Apache/Tomcat combinations, as the Apache/Tomcat solution provides basic load balancing functionality as part of an economical Open Source solution. 23

27 How Does This Configuration Work? Rather than immediately delve into the configuration file specifics necessary to implement the sample configuration, a quick explanation of what is setup provides a quick introduction to how such an environment works. Machine A The Load Balancer The first machine in the example provides the interface to the Internal/Intranet and makes the decisions as to how to distribute HTTP requests to the different instances of EKP. In the case of Apache 2.0, this is accomplished using a special load balancing worker process that decides which Tomcat to forward each request to this load balancing worker (labeled as LBW in the diagram) actually makes use of the real mod_jk workers that communicate with each Tomcat instance, hence its function is more of a local coordinator/router than a direct communications path to each Tomcat. By default, this load balancer uses sticky sessions, meaning that a user who initially is routed to a specific Tomcat instance has all subsequent requests routed to the same Tomcat. This is important for EKP, as each user session uses a local cached user object that should be maintained throughout the user session. This Apache instance must also reference a shared version of the /nd/fresco web site directory, as it is Apache (not Tomcat or EKP) that actually retrieves the gifs, jpgs, html pages, course resources, and files that make up a complete site (more on this is a subsequent section). So, for this sample configuration, you must decide where to put these files. The basic choices are: On MachineA On MachineD On a network addressable storage system (NAS/SAN). Note that the courseware is likely to be stored here as well, although even if you are using separate Content Servers, you still need to have a shared /nd/fresco web directory. This machine may be replaced by a special-purpose hardware device whose job it is to route request requests to the multiple Tomcat instances, in which case none of the Apache or worker configuration discussions that are provided below apply, and each Tomcat instance (instead of Apache) will retrieve the gifs, jpegs, html pages, and files from the shared /nd/fresco web directory. Machine B Tomcat1/EKP1 This machine contains a normal instance of Tomcat and of EKP. However, EKP must identify itself with special node ID properties (discussed later) that allow it to avoid conflict with other EKP instances going against the same database. This is done via two properties in ekp.properties: system.nodeid=1 (the default value) system.nodename=ekp (the default value) 24

28 where system.nodename is a 3-character identifier used by EKP to construct unique object keys. In addition, Tomcat must reference a shared version of the /nd/fresco web site directory so that any uploaded files/courses are also available to other EKP instances. Machine C Tomcat2/EKP2 This machine also contains a normal instance of Tomcat and of EKP. The node identity of this instance must be unique: system.nodeid=2 system.nodename=abc This Tomcat instance must also reference a shared version of the /nd/fresco web site directory. Machine D Database Server This machine contains a database instance that is accessed by all EKP instances. This is a critical component in the overall solution, and this machine is often a separate cluster to ensure that a failure in the primary DB server does not bring down the entire service. It is certainly technically possible to put the database server on one of the other machines, but this solution is suitable only for economical load balancing, but not resiliency. Basic Requirements How many machines you use, and which services you put on which machine, is a matter of your objectives is the emphasis on scalability, resiliency, or cost. It is fairly easy to configure Apache/Tomcat into a Load Balancing configuration that is scalable to a large number of users if you are not concerned about resiliency, as you can simply add nodes as the site grows. Resiliency is more complicated, as you must also provide redundancy for the database engine as well as the Load Balancer/Dispatcher mechanism, otherwise you have single points of failure in the configuration. What is more, different databases and applications servers have different capabilities in this area. For example, Oracle supports Distributed Processing with multiple nodes accessing the same database files at the same time, whereas Sybase and Microsoft support Warm Standby modes in which there must be a fail-over to another server instance to continue processing. In a nutshell, for the different types of configurations available, the following issues must be addressed: 25

29 1. Make the web directory shared; 2. Setup a machine to function as a load balancer. For Apache, this is done by configuring the contents of the httpd.conf, mod_jk.conf, and worker.properties files. 3. Identify each EKP with its unique node ID and node name in ekp.properties, plus cache handling. 4. Configure the selected application server (Tomcat, in our example) according to its capabilities (they are not all the same in this area, although most fundamental capabilities tend to be supported); Prior to making any configuration decisions, ensure that your objectives are clearly identified (priority on scalability versus fault tolerance) and decide how best to address single-point failure issues for the load balancer/dispatcher, web site, and the database server given your HW and software constraints. STEP 1 -- Shared Web Site Folder As mentioned above, the web site directory needs to be shared so that it is accessible by Apache as well as each Tomcat/EKP instance. This is necessary because the LMS site, unlike most web sites, is not a static set of files. In particular: Apache actually retrieves (and caches) the files (course content, gifs, repository documents, etc.) for delivery to the user, so it needs access to this folder. Each Tomcat instance is functioning primarily as a servlet engine for executing business logic, whereas Apache is an efficient web file retrieval engine. Each Tomcat/EKP instance may create/upload new files in the web site directory in the following situations: o o o o o Users upload homework; Users/instructors add files to the repository; Administrators uploads new skins or user loader CSV files; Users run reports for CSV file download; Content packages are uploaded; For the purposes of this exercise, it is assumed that a shared drive Z:\shared has been created which contains the following EKP web site files and directories: /shared /nd/ index.html help.html (an arbitrary name used for our example) (The bulk of the EKP web site) (The login page related files) 26

30 lostpass.html /images These are the same web files you find in the <Tomcat>/webapps/ekp that makes up a standard non-clustered installation. The Tomcat/EKP specific directories (e.g. <Tomcat>/webapps/ekp/WEB-INF/.. ) will remain with each Tomcat instance as these contain the servlet and EKP configuration files and are used to startup EKP on each node. STEP 2 -- Setting up Apache There are three sets of files that need to be configured for Apache: The standard../conf/httpd.conf The../conf/jk/mod_jk.conf The../conf/jk/work.properties The <apache>/conf/httpd.con file This file defines, among other things, the basic URL and hosts supported by this Apache instance. For out example, we need to add the following (assuming that out host URL is sample.netdimensions.com) to the end of this file: ##### sample.netdimensions.com ##### <VirtualHost :80> (put your IP address here) ServerAdmin support@netdimensions.com DocumentRoot "z:/shared" ServerName sample.netdimensions.com ErrorLog logs/sample-error.log CustomLog logs/sample-access.log common </VirtualHost> <Directory z:/shared > Options FollowSymLinks AllowOverride None Order allow,deny Allow from all 27

31 </Directory> #Include the mod_jk configuration Include "c:\program Files\Apache Group\Apache2\conf\jk\mod_jk.conf" This configuration specification would allow you to reference the EKP cluster using an URL of the following type: The important parts of our changes here are that we identify a shared folder for the document root, and we direct Apache to the mod_jk configuration files. Note that the actual mod_jk.dll executable must also be placed in the <Apache>/Modules folder so that the workers and load balancer can be loaded by Apache at startup time. <apache>/conf/jk/mod_jk.conf The file mod_jk.conf is read by Apache to define the Tomcat mappings to be used for each application. Since Apache is on a separate machine from Tomcat(s), it needs to have its own copy of the <jk> configuration files (for example, <Apache>\conf\jk\mod_jk.conf and <Apache>\conf\jk\worker.properties ). These files are usually put in the Tomcat folder, but since that is on another machine we simply move it to <Apache>/conf/jk as a convenience. Note that Apache sees a single logical Tomcat worker (lbrouter) no matter how many instances of EKP you run. You could also include direct access to each EKP instance if there were some scenarios where you want to directly access a specific machine. This configuration is not covered here as the immediate goal is to just describe the load balancing requirements without overcomplicating the configuration files. Note Note that Apache sees a single logical Tomcat worker (lbrouter) no matter how many instances of EKP you run. You could also include direct access to each EKP instance if there were some scenarios where you want to directly access a specific machine. This configuration is not covered here as the immediate goal is to just describe the load balancing requirements without overcomplicating the configuration files. For more information on how to setup these files, please refer to 28

32 # Static files Alias /ekp "z:/shared" <Directory "z:/shared"> Options -Indexes FollowSymLinks DirectoryIndex index.html index.htm index.jsp </Directory> Alias /nd "Z:/shared/ekp/nd" <Directory "Z:/shared/ekp/nd"> Options -Indexes FollowSymLinks DirectoryIndex index.html index.htm index.jsp </Directory> JkMount /ekp/tx/* lbrouter JkMount /ekp/servlet/* lbrouter JkMount /ekp/*.jsp lbrouter JkMount /ekp/servlet/ekp/* lbrouter JkMount /ekp/*.tx lbrouter../conf/jk/worker.properties The worker.properties file is used to define connections/channels between Apache and Tomcat. Since we are going to be running multiple Tomcats, we need to define multiple or ajp13 workers, plus a load balancing worker. A sample worker.properties file would look like this: # BEGIN workers.properties worker.list=lbrouter # Definition for Ajp13 worker for Tomcat Instance #1 worker.lb1.port=8109 worker.lb1.host=machinea.netdimensions.com worker.lb1.type=ajp13 worker.lb1.lbfactor=1 worker.lb1.local_worker=0 # Definition for Ajp13 worker for Tomcat Instance #2 worker.lb2.port=8209 worker.lb2.host=machineb.netdimensions.com worker.lb2.type=ajp13 worker.lb2.lbfactor=1 worker.lb2.local_worker=0 29

33 # Definition for Load Balancer worker.lbrouter.type=lb worker.lbrouter.balanced_workers=lb1,lb2 worker.lbrouter.local_worker_only=0 # END workers.properties There are numerous other load balancing tuning options not covered in this document. The reader is referred to for more details. The ones shown above are sufficient for our needs in this example. STEP 3 -- Setting up Tomcat Setting up Tomcat on each node is relatively simple as each instance is identical except for the following changes in the server.xml file. 1. Modify the ajp13 worker port so that it corresponds to the port values used in the workers.properties file for that Tomcat instance. For example:. <!-- Define a Coyote/JK2 AJP 1.3 Connector on port > <Connector classname="org.apache.coyote.tomcat4.coyoteconnector" port="8109" minprocessors="5" maxprocessors="75" enablelookups="true" redirectport="8443" acceptcount="10" debug="0" connectiontimeout="20000" useurivalidationhack="false" protocolhandlerclassname="org.apache.jk.server.jkcoyotehandler"/> 2. Modify the jvmroute parameter of the <Engine> directive so that the load balancer will use session affinity. This is critical for EKP!. For example: <Engine name="standalone" defaulthost="localhost" jvmroute="lb1"> The name used here, LB1 is unique for each Tomcat instance and must match the worker name as found in worker.properties. You will find Apache documentation is very thin on this point, but trust that it is essential for the session affinity to work properly. 30

34 Sample Tomcat server.xml Configuration File Below is a sample server.xml configuration file for a multiple instance Tomcat environment. In this file: Other unused items have been eliminated (e.g. no HTTP port listener); The AJP13 port (8109 in this example) must be unique for each instance. The shutdown port (8105 in this example), if used, must be unique for each instance This example is just that an example. You may add other items as your needs dictate. <!-- Example Server Configuration File --> <!-- Note that component elements are nested corresponding to their parent-child relationships with each other --> <!-- A "Server" is a singleton element that represents the entire JVM, which may contain one or more "Service" instances. The Server listens for a shutdown command on the indicated port. --> Note: A "Server" is not itself a "Container", so you may not define subcomponents such as "Valves" or "Loggers" at this level. <Server port="8105" shutdown="shutdown" debug="0"> <!-- Uncomment these entries to enable JMX MBeans support --> <Listener classname="org.apache.catalina.mbeans.serverlifecyclelistener" debug="0"/> <Listener classname="org.apache.catalina.mbeans.globalresourceslifecyclelistener" debug="0"/> <!-- Global JNDI resources --> <GlobalNamingResources> <!-- Test entry for demonstration purposes --> <Environment name="simplevalue" type="java.lang.integer" value="30"/> <!-- Editable user database that can also be used by UserDatabaseRealm to authenticate users --> <Resource name="userdatabase" auth="container" type="org.apache.catalina.userdatabase" description="user database that can be updated and saved"> </Resource> <ResourceParams name="userdatabase"> <parameter> <name>factory</name> <value>org.apache.catalina.users.memoryuserdatabasefactory</value> </parameter> <parameter> <name>pathname</name> <value>conf/tomcat-users.xml</value> </parameter> </ResourceParams> 31

35 </GlobalNamingResources> <!-- A "Service" is a collection of one or more "Connectors" that share a single "Container" (and therefore the web applications visible within that Container). Normally, that Container is an "Engine", but this is not required. --> Note: A "Service" is not itself a "Container", so you may not define subcomponents such as "Valves" or "Loggers" at this level. <!-- Define the Tomcat Stand-Alone Service --> <Service name="tomcat-standalone"> <!-- Define a Coyote/JK2 AJP 1.3 Connector on port > <Connector classname="org.apache.coyote.tomcat4.coyoteconnector" port="8109" minprocessors="5" maxprocessors="100" enablelookups="true" redirectport="8143" acceptcount="15" debug="0" connectiontimeout="20000" useurivalidationhack="false" protocolhandlerclassname="org.apache.jk.server.jkcoyotehandler"/> <!-- An Engine represents the entry point (within Catalina) that processes every request. The Engine implementation for Tomcat stand alone analyzes the HTTP headers included with the request, and passes them on to the appropriate Host (virtual host). --> <!-- Define the top level container in our container hierarchy --> <Engine name="standalone" defaulthost="localhost" debug="0"> <!-- Global logger unless overridden at lower levels --> <Logger classname="org.apache.catalina.logger.filelogger" prefix="catalina_log." suffix=".txt" timestamp="true"/> <!-- Because this Realm is here, an instance will be shared globally --> <!-- This Realm uses the UserDatabase configured in the global JNDI resources under the key "UserDatabase". Any edits that are performed against this UserDatabase are immediately available for use by the Realm. --> <Realm classname="org.apache.catalina.realm.userdatabaserealm" debug="0" resourcename="userdatabase"/> <!-- Define the default virtual host --> <Host name="localhost" debug="0" appbase="webapps" unpackwars="true" autodeploy="true"> a <!-- Logger shared by all Contexts related to this virtual host. By default (when using FileLogger), log files are created in the "logs" directory relative to $CATALINA_HOME. If you wish, you can specify a different directory with the "directory" attribute. Specify either relative (to $CATALINA_HOME) or absolute path to the desired directory.--> 32

36 <Logger classname="org.apache.catalina.logger.filelogger" directory="logs" prefix="localhost_log." suffix=".txt" timestamp="true"/> <!-- Define properties for each web application. This is only needed if you want to set non-default properties, or have web application document roots in places other than the virtual host's appbase directory. --> </Host> </Engine> </Service> </Server> 33