Spectrum Technology Platform Version 8.0.0 Tutorial: Load Balancing Spectrum Spatial Services UNITED STATES www.pb.com/software Technical Support: www.pbinsight.com/support CANADA www.pb.com/software Technical Support: www.pbinsight.com/support EUROPE/UNITED KINGDOM www.pitneybowes.co.uk/software Technical Support: www.pbsoftware.eu/uk/support ASIA PACIFIC/AUSTRALIA www.pitneybowes.com.au/software Technical Support: www.pbinsight.com.au/support The Spectrum spatial services that are included with the Location Intelligence Module are designed to support enterprise scale deployment requirements. This includes deployment configurations for high availability and support for horizontal and vertical scaling. Spectrum's high availability configuration enables organizations to create fault tolerant deployments supporting requirements for continuous service provision. Spectrum s configurations for horizontal and vertical scaling provide the ability to grow the capacity of a system to support greater load. The horizontal scaling configuration supports additional load through addition of more servers to the cluster. The vertical scaling configuration grows the capacity of the system to support larger loads through the addition of greater hardware resources. Spectrum uses a load balancing based approach to deliver resilience and horizontal scaling architectures. This tutorial will illustrate how to setup a load balanced Spectrum. Contents: About this Tutorial.......................................2 Deployment Architecture..................................2 Install Spectrum..........................................3 Set Up a Map Image File Share..............................4 Configure Spectrum.......................................4 Configure Common Repository.............................5 Shared Spectrum Local Data...............................9 Spectrum Performance Tuning..............................9 Set Up Load Balancer....................................10 2012 Pitney Bowes Software Inc. April 04, 2013
About this Tutorial About this Tutorial The goal of this tutorial is to illustrate the concepts of load balancing a Spectrum installation with the spatial services for resilience or high capacity. Both the concepts and the steps outlined in this tutorial can apply to virtualized and native environments. The tutorial will provide information on scaling spatial server horizontally and vertically. This tutorial uses the following hardware/software and Spectrum components: Spectrum 8.0 LIM Mapping Service PostGres database 9.0.3 RedHat Linux 5.4 Deployment Architecture In this tutorial we will create a load balanced Spectrum deployment. The diagram below illustrates the deployment architecture of the configuration we will create. Load balancing can be used to support high availability and scaling. The deployment architecture includes a load balancer, Spectrum spatial services cluster, database and a file share. With this approach it is possible to scale both horizontally and vertically. Load Balancer The load balancer spreads requests between the Spectrum instances. Any load balancer that supports load balancing HTTP/HTTPs requests can be used. Spectrum Cluster The cluster is a collection of Spectrum instances with LIM sharing administration, named resources, geographical metadata content and configuration settings. Additional nodes can be added to the cluster for resilience or to deliver support for greater loads. Each node can be scaled vertically through additional hardware resources and/or additional instances should this be required for hardware with massive resources. Spectrum can be configured to use restricted numbers of CPUs. 2 Spectrum Technology Platform 8.0.0
Spectrum Spatial Services Load Balance Tutorial Database Spectrum stores named resources (maps, layers, tables and styles), geographic metadata and configuration in a repository. In the default single server installation an embedded database is used to store these resources on the local server. To create a resilient scalable solution this embedded database should be replaced with a resilient independent database. In this release Spectrum supports Oracle, PostGreSQL(PostGIS) and Microsoft SQL Server as repository databases. In the load balanced configuration Spectrum nodes cache these resources in a local cache and search index in each node in the cluster. When a Spectrum node receives a request it uses the local cache and index to find resources. Named resources can be added through any node in the cluster. Each node keeps its cache current by checking for differences between its local cache and the central database. This check occurs every 2 seconds by default. Time frequency can be configured. This architecture ensures the server delivers high performance transactions and the load on the repository database is kept to a minimum. If a new Spectrum is added to the cluster the cache and index are created automatically. Such a scenario can occur to remedy a node failure or grow the capability of the deployment. File Share The file share provides a folder to hold map images generated by Spectrum. When maps are renderer using the web services the server supports the map images being returned through URLs or returned as a base 64 encoded image. When a URL is returned the map image is stored as a file and served on request of the URL. To ensure any Spectrum node can return the map image a file share is used to store the images. Install Spectrum In this step we will create a Spectrum deployment. To install Spectrum into your VM instance: 1. Copy the Spectrum Linux installer to the target server with Red Hat operating system. 2. Install Spectrum. For this tutorial we will follow the default installation but not install the license key. The installation guide for UNIX and Linux provides more information on the installation process. a) Locate the install.sh installer b) Ensure the user has execute permission chmod a+x install.sh c) Execute install.sh install.sh The installer will walk you through the installation process. 3. Copy the license key into the server/app/import directory of each Spectrum installed. 4. Start Spectrum It is possible to install multiple installations of Spectrum on the same operating system. This can be used to provide flexibility when vertically scaling the server. To support multiple installations on the Tutorial: Load Balancing Spectrum Spatial Services 3
Set Up a Map Image File Share same machine the hidden file /var/.com.zerog.registry.xml needs to be renamed to enable a new installation to a different folder and port on the same machine. Spectrum is now installed. The next step is to configure the Spectrum load balanced cluster deployment. Set Up a Map Image File Share The file share provides a folder to hold map images generated by Spectrum. Create a shared folder accessible to all Spectrum nodes. The file share is not required if maps are returned from the web services as base 64 encoded images. To set up a map image file share: 1. Task s Mount a shared folder on each operating system hosting Spectrum. The commands below mount a drive on a Microsoft Windows Server or network drive supporting CIFS. mkdir /mnt/<linux mount> mount -t cifs //<windows host>/<windows share> /mnt/<linux mount>-o username=shareuser,password=sharepassword,domain=pbi 2. Set the image share to load at startup in /etc/fstab. //<windows ip address for share>/share /path_to/mount cifs username=server_user,password=secret,_netdev 0 0 Configure Spectrum Once Spectrum is installed, you need to configure your instance before you can replicate it to another virtual machine. If you are not using a virtual machine environment, you will need to perform these steps on each of your Spectrum installations. Add the Map File Share to Spectrum Context for the current task 1. Modify the Mapping service configuration by pointing to a shared image folder and load balance server. In the ImageCache change the Directory parameter to a common image directory, and change the AccessBaseURL parameter to the load balancer machine image URL. If using a virtual machine environment, remember this IP address, as you must set the load balancer VM to this IP in the section Set Up Load Balancer on page 10. <ImageCache> <Directory>/mnt/<linux mount>/images</directory> <AccessBaseURL>http://<loadbalance_IP_address>/Spatial/images</Access- BaseURL> <FileExpire>30</FileExpire> <ScanInterval>30</ScanInterval> </ImageCache> 4 Spectrum Technology Platform 8.0.0
Spectrum Spatial Services Load Balance Tutorial 2. Set up symbolic link to enable map images to go to the shared file system. cd /<spatial server root>/server/modules/spatial rm Rf images ln -s / mnt/<linux mount>/images Modify the Service Configurations To modify the service configurations for load balancing: In each service configuration file, change the <RepositoryURL> to point to the load balance server repository URL. The RepositoryURL should change to point to the balancer from http://<spectrum>/repositoryservice/rmi to http://<balancer>/repositoryservice/rmi. Modify Java Properties File To modify the java properties for Spectrum Technology Platform: 1. Modify the java.properties file, located in <spectrum>/server/modules/spatial/java.properties, to point to the load balance server. 2. Change the images.webapp.url and all of the service host and port numbers to point to the load balance server. Configure Ports for Multiple Spectrum Instances If you have multiple Spectrum Technology Platform instances on a single machine, you must change the port numbers. To change the port numbers for each Spectrum instance: 1. Change all ports in <Spectrum root>/server/app/conf/spectrum-container.properties to new port values that are not in use. The http port reflects the port number entered in the installer. 2. Update the rmi port in bootstrap.properties in the bin/jackrabbit folder (e.g. 11099). The default is 1099. Configure Common Repository The next step is to configure Spectrum to use a common repository database for the cluster. This ensures that named resources, geographic metadata and configuration settings are managed across the cluster. The description below describes how to configure Spectrum to use a common repository database. This tutorial will use a PostgreSQL database. It is possible to use an Oracle or Microsoft SQL Server database. The repository is installed with a set of named resources, geographic metadata and configuration files. To migrate these resources to the common database repository the resources need to be exported from the default internal repository database and reimported into the new shared repository database. Tutorial: Load Balancing Spectrum Spatial Services 5
Bulk Export Using WebDAV To provide support for bulk export/import of repository content Spectrum repository provides a WebDAV interface. Bulk Export Using WebDAV You need to export the contents of the installed repository. This step only needs to be performed once as the contents of the repository should be the same at this point for all instances of Spectrum. There are many free WebDAV clients including Microsoft Windows Explorer and GNOME- Nautilus. For this Linux focused tutorial we will use GNOME- Nautilus to export the resources from Spectrum. 1. Start Spectrum. 2. Connect to Spectrum using WebDAV: a) Connect to the WebDAV Directory using GNOME Nautilus. b) Select Connect to Server in the Places menu. This will open a File Browser window. The repository is at the following location. http://<yourmachine>:<port>/repositoryservice/repository/default 3. Copy the content of the repository to a local drive. Set Up the Common Repository Database These steps need to be performed on all instances of Spectrum in your load balanced environment: 1. Stop Spectrum. 2. Add the database JDBC drivers to the Spectrum common lib directory to allow it to use the selected database. In this tutorial we are using PostGreSQL JDBC drivers. Copy the /<spectrum root>/server/modules/spatial/lib/postgresql-8.4-701.jdbc4.jar file to /<spectrum root>/server/app/lib/postgresql-8.4-701.jdbc4.jar 6 Spectrum Technology Platform 8.0.0
Spectrum Spatial Services Load Balance Tutorial 3. Edit the /<spectrum root>/server/modules/spatial/jackrabbit/repository.xml file to point the repository to a database and add clustering. There are four separate changes you need to make: a) Modify the two FileSystem sections within the Repository and Workspace sections of the file: <FileSystem class="org.apache.jackrabbit.core.fs.db.dbfilesystem"> <param name="driver" value="org.postgresql.driver"/> <param name="url" value="jdbc:postgresql://<hostname>:<port>/<databasename>"/> <param name="schema" value="postgresql"/> <param name="user" value="<user>"/> <param name="password" value="<pwd>"/> <param name="schemaobjectprefix" value="rep_"/> </FileSystem> b) Modify the Persistence Manager within the Workspace: <PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.postgresqlpersistencemanager"> <param name="url" value="jdbc:postgresql://<hostname>:<port>/<databasename>"/> <param name="schema" value="postgresql"/> <param name="user" value="<user>"/> <param name="password" value="<pwd>"/> <param name="schemaobjectprefix" value="${wsp.name}_"/> <param name="externalblobs" value="false"/> </PersistenceManager> c) Enable Clustering at the end of the file, right above the </Repository> tag. Each instance of Spectrum will need to have a distinct id to enable synchronization of clustering to work. The delay defines the time delay for synchronization in milliseconds. <Cluster id="node1" syncdelay="2000"> <Journal class="org.apache.jackrabbit.core.journal.databasejournal"> <param name="revision" value="${rep.home}/revision.log" /> <param name="driver" value="org.postgresql.driver" /> <param name="url" value="jdbc:postgresql://<hostname>:<port>/<databasename>" /> <param name="schema" value="postgresql"/> <param name="schemaobjectprefix" value="rep_"/> <param name="user" value="<user>"/> <param name="password" value="<pwd>"/> <param name="databasetype" value="postgresql"/> </Journal> </Cluster> d) Comment out the DataStore section: <DataStore class="org.apache.jackrabbit.core.data.filedatastore"/> 4. Remove the following folders from the /server/modules/spatial/jackrabbit directory for each instance of Spectrum: repository, version, workspaces. 5. If your PostGIS database has previously had repository content added, you must remove tables from your database so a clean repository can be created. If you are starting with a new database, please make sure the tables do not exist. The following tables need to be removed from the database: public.default_names_id_seq public.default_binval public.default_bundle public.default_names public.default_refs Tutorial: Load Balancing Spectrum Spatial Services 7
Import the Repository Content public rep_fsentry public.rep_global_revision public.rep_journal public.rep_local_revisions public.security_binval public.security_bundle public.security_names public.security_refs Import the Repository Content Next, import the content of the repository we previously exported back into the repository. This step only needs to be performed on one of the Spectrum instances. 1. Start Spectrum 2. Copy the previously exported content of the repository back into the repository. First, connect to Spectrum using WebDAV: a) Connect to the WebDAV Directory using GNOME Nautilus. Select Connect to Server... in the Places menu. This will open a File Browser window. The repository is at the following location. http://<yourmachine>:<port>/repositoryservice/repository/default. b) Copy the content to the repository root directory. Spectrum is now configured and ready to be load balanced. 8 Spectrum Technology Platform 8.0.0
Spectrum Spatial Services Load Balance Tutorial Shared Spectrum Local Data If you are using TAB file data on the file system, this data needs to be in a shared location accessible by all instances of Spectrum in the load balanced environment. It is also important to note that all named resources in the repository accessing data on the file system should point to this shared location. Each VM or machine hosting Spectrum needs to have access to the mounted shared drive. Note: Using named resources that point to database tables do not require a shared drive, as the named resources in the repository do not access the data using a file path; rather they use a named connection to the data in the database. Spectrum Performance Tuning Spectrum provides several tuning options to optimize performance of the server. The optimal selection of settings is dependent on the nature of the deployment. To create a well-tuned server environment, it is recommended that performance tests should be executed in the deployed environment to determine optimal settings. This section provides some general guidance on performance tuning. More information on tuning can be found in the Performance Tuning Guide. JVM Tuning Spectrum is a Java server, and as a result, JVM tuning parameters can be used to optimize performance. The JVM can be configured through the /<spectrum root>/server/modules/spatial/java.vmargs file. To optimize Spectrum's performance using JVM tuning parameters: 1. Open this file in a text editor. Set the maximum memory allocation on the JVM. An allocation of 512m for each active CPU core is generally appropriate. Remember not to exceed the maximum memory available to your operating system and leave a suitable space for the operating system to do its work. In this tutorial set the max we will set it to 4GB using the setting -Xmx 4096M leaving 1GB for the operating system. Save the file. Restart the Spectrum. 2. Save the file. 3. Restart Spectrum. Additional JVM tuning can be used with Spectrum. Remote Component Configuration Each of spatial service component in Spectrum is deployed into its own JVM instance separate from the Spectrum Platform run time. This ensures the platform is independent of the modules within it and JVM configuration can be applied per service allowing flexibility of memory allocation and tuning for performance based on the characteristics of the service. Tutorial: Load Balancing Spectrum Spatial Services 9
Tuning the Components Using MaxActive Setting Tuning the Components Using MaxActive Setting In addition to JVM tuning, the components can be tuned using the MaxActive setting. This setting represents the number of threads on the components listening for service requests from the Spectrum Platform. Every web service request enters Spectrum from the platform and is passed to the components This value should be increased to accommodate greater request loads. Performance test should be run with various settings until optimal performance is achieved for the usage. The maximum setting should not go above twice the number of CPU core. The Spectrum platform provides capability for tuning remote components using the JMX console (http://host:port/jmx-console/). The components appear in the Domain: com.pb.spectrum.platform.config.component section as follows: com.pb.spectrum.platform.config.component:pool=spatial.feature.default com.pb.spectrum.platform.config.component:pool=spatial.mapping.default com.pb.spectrum.platform.config.component:pool=spatial.geometry.default com.pb.spectrum.platform.config.component:pool=spatial.maptiling.default Set Up Load Balancer Now that multiple instances of Spectrum are deployed and configured, the instances need to be load balanced. In this tutorial Apache HTTP Server is used as a load balancer. Any load balancer with support for load balancing http requests can be used. In the Apache HTTP Server configuration (httpd.conf file e.g., /etc/httpd/conf/) turn on the proxy and load balance modules by adding the following sections at the end of the file. You will need to add the IP addresses for the number of Spectrum instances you have created (replace <VMCloneIP1>:<Port Number> with the correct IP address for each VM or system address. <IfModule proxy_module> ProxyRequests Off <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPreserveHost On #Load Balancer Manager <Location /balancer-manager> SetHandler balancer-manager Order deny,allow Allow from all </Location> #MappingService <Proxy balancer://spatial > BalancerMember http://<vmcloneip1>:8080/spatial BalancerMember http://<vmcloneip2>:8080/spatial BalancerMember http://<vmcloneip3>:8080/spatial BalancerMember http://<vmcloneip4>:8080/spatial </Proxy> ProxyPass /Spatial balancer://spatial 10 Spectrum Technology Platform 8.0.0
#MappingService <Proxy balancer://soap> BalancerMember http://<vmcloneip1>:8080/soap BalancerMember http://<vmcloneip2>:8080/soap BalancerMember http://<vmcloneip3>:8080/soap BalancerMember http://<vmcloneip4>:8080/soap </Proxy> ProxyPass /soap balancer://soap #RepositoryService <Proxy balancer://repositoryservice> BalancerMember http://<vmcloneip1>:8080/repositoryservice BalancerMember http://<vmcloneip2>:8080/repositoryservice BalancerMember http://<vmcloneip3>:8080/repositoryservice BalancerMember http://<vmcloneip4>:8080/repositoryservice </Proxy> ProxyPass /RepositoryService balancer://repositoryservice #REST Service <Proxy balancer://rest> BalancerMember http://<vmcloneip1>:8080/rest BalancerMember http://<vmcloneip2>:8080/rest BalancerMember http://<vmcloneip3>:8080/rest BalancerMember http://<vmcloneip4>:8080/rest </Proxy> ProxyPass /rest balancer://rest </IfModule> For load balancing Spectrum, the following modules need to be loaded. LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_ftp_module modules/mod_proxy_ftp.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_connect_module modules/mod_proxy_connect.so You do not need to perform this task for load balancing the Apache HTTP Server that ships with the CentOS. These modules are loaded by default. Once you have updated your conf file save the changes and restart the load balancer. Use the following command to restart the Apache HTTP Server load balancer: service httpd start chkconfig httpd on Spectrum Spatial Services Load Balance Tutorial Note: Sending a request through the load balance server might time out if the firewall of the Spectrum instances are turned on. In this scenario, either turn off the firewall on these machines, or modify the service configuration files (e.g., MappingConfiguration) to point to the Spectrum instance for that machine. For instance change the RepositoryURL from the load balance server http://<balancer>/repositoryservice/rmi to the local Spectrum instance http://<spectrum>/repositoryservice/rmi. Tutorial: Load Balancing Spectrum Spatial Services 11