Webthority 6.6 Best Practice Guide
2012 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of Quest Software, Inc. If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 USA www.quest.com email: legal@quest.com Refer to our Web site for regional and international office information. TRADEMARKS Quest, Quest Software, the Quest Software logo are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Software's trademarks, please see http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered trademarks are property of their respective owners. Disclaimer The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. Product Name: Webthority Best Practice Guide Updated - April 2012 Software Version - 6.6
CHAPTER 1 ABOUT WEBTHORITY................................ 5 AUDIENCE AND SCOPE............................ 6 CONVENTIONS................................. 7 ABOUT QUEST SOFTWARE.......................... 8 CONTACTING QUEST SOFTWARE...................... 8 CONTACTING CUSTOMER SUPPORT.................. 8 CHAPTER 2 WEBTHORITY BEST PRACTICE.......................... 9 RECOMMENDED CONTENT MAPPING CONFIGURATIONS.........10 1 - ROOT TO ROOT MAPPING.....................10 2 - SYMMETRIC VIRTUAL DIRECTORY MAPPING..........11 FURTHER READING...........................11 OPTIMIZING WEBTHORITY FOR A PRODUCTION ENVIRONMENT....12 BACKGROUND..............................12 MEMORY.................................12 HTTP CONNECTIONS..........................13 LOGGING.................................14 ADDITIONAL CONSIDERATIONS....................14 TROUBLESHOOTING...........................15 WEBTHORITY BACKUP AND RESTORE...................16 BACKUP..................................16 RESTORE.................................18 3
About Webthority Introduction to Webthority Audience and Scope Conventions About Quest Software Contacting Quest Software 5
Webthority Overview Webthority is a component of the Quest One Identity Solution, a set of enabling technologies, products, and integration that empowers organizations to simplify identity and access management by: Reducing the number of identities Automating identity administration Ensuring the security of identities Leveraging existing investments, including Microsoft Active Directory. Quest One improves efficiency, enhances security and helps organizations achieve and maintain compliance by addressing identity and access management challenges as they relate to: Single sign-on Directory consolidation Provisioning Password management Strong authentication Privileged account management Audit and compliance. Audience and Scope This book provides Webthority best practice guidance for administrators who want to install and configure Webthority, This book does not provide tutorial information on the use of the Windows operating system or on network communication concepts. Users must have experience in using the specified operating system and an understanding of networking concepts. 6
Webthority Overview Conventions In order to help you get the most out of this guide, we have used specific formatting conventions. These conventions apply to procedures, icons, keystrokes, and cross-references. ELEMENT Select Bolded text courier text Italic text Bold Italic text Blue text CONVENTION This word refers to actions such as choosing or highlighting various interface elements, such as files and radio buttons. Used to highlight installation questions and responses. File, daemon, utility, option, attribute names. Used for comments. Used for emphasis. Indicates a cross-reference. When viewed in Adobe Acrobat, this format can be used as a hyperlink. Used to highlight additional information pertinent to the process being described. Used to provide Best Practice information. A best practice details the recommended course of action for the best result. Used to highlight processes that should be performed with care. + A plus sign between two keystrokes means that you must press them at the same time. A pipe symbol (vertical bar) between elements means that you must select the elements in that particular sequence. \ The back slash, immediately followed by a new line, indicates a Unix command line continuation. <version>.<build number> References to the product version you are installing are displayed with <version>.<build number> in angle brackets. 7
Webthority Overview About Quest Software Quest Software, Inc., a two-time winner of Microsoft s Global Independent Software Vendor Partner of the Year award, delivers innovative products that help organizations get more performance and productivity from their applications, databases Windows infrastructure and virtual environments. Through a deep expertise in IT operations and a continued focus on what works best, Quest helps more than 100,000 customers worldwide meet higher expectations for enterprise IT. Quest s Windows management solutions simplify, automate secure and extend Active Directory, Exchange Server, SharePoint, SQL Server,.NET and Windows Server as well as integrating Unix, Linux and Java into the managed environment. Quest Software can be found in offices around the globe and at www.quest.com. Contacting Quest Software Phone Email Mail 949.754.8000 (United States and Canada) info@quest.com Quest Software World Headquarters 5 Polaris Way Aliso Viejo, CA 92656 Web site www.quest.com Please refer to our Web site for regional and international office information. Contacting Customer Support Quest Software's world-class support team is dedicated to ensuring successful product installation and use for all Quest Software solutions. 8 SupportLink www.quest.com/support Email at support@quest.com You can use SupportLink to do the following: Create, update, or view support requests Search the knowledge base Access FAQs Download patches
Webthority Best Practice Recommended Content Mapping Configurations Optimizing Webthority for a Production Environment Webthority Backup and Restore 9
Recommended Content Mapping Configurations For complex sites such as Microsoft SharePoint and Microsoft OWA where a large proportion of the content is dynamic, Quest recommends setting up the content mappings in one of the two configurations outlined below to ensure the path portion of the URL is not altered. 1 - Root to Root Mapping This is the simpler of the two configurations. The external FQDN maps directly to the internal FQDN of the content server, without specifying the path portion of the URL, for example: https://mail.democorp.com:443 -> http://mail.internal.local:80 Where multiple content servers are required on a single Webthority proxy, additional external FQDN s can be created which also resolve to the same IP address of the Webthority proxy, for example: https://mail.democorp.com:443 https://sharepoint.democorp.com:443 https://qpm.democorp.com:443 -> http://mail.internal.local:80 -> http://sharepoint.internal.local:80 -> http://qpm.internal.local:80 In this situation, a single SSL certificate should be purchased or created to cover all external FQDN's. This can be done either via a wildcard certificate (*.democorp.com) or a Subject Alternative Name (SAN) certificate. Refer to How To Generate a Certificate Covering Multiple FQDN s in the Webthority guide entitled How To Obtain a Signed SSL Certificate. 10
2 - Symmetric Virtual Directory Mapping If the content servers each house their applications within a unique virtual directory (for example http://qpm.internal.local:80/qpm and http://ars.internal.local:80/arserver) the mappings can be configured symmetrically using a single external FQDN. For example: https://idm.democorp.com:443/qpm https://idm.democorp.com:443/arserver -> http://qpm.internal.local:80/qpm -> http://ars.internal.local:80/arserver When using this configuration take care to ensure that the path portion of the URL is not altered between the proxy URL and the content URL, even down to subtle changes such as character case. Further Reading For further information on how to configure Microsoft Outlook Web Access and Microsoft SharePoint, please refer to the following guides: Webthority 6.6 How To Proxy Microsoft Outlook Web Access Webthority 6.6 How To Proxy Microsoft Sharepoint. 11
Optimizing Webthority for a Production Environment Before deploying Webthority into production we recommend that you perform the following steps to ensure Webthority is optimized to handle the expected workload. Background Typically a user s browser will use two persistent HTTP connections (http://en.wikipedia.org/wiki/http_persistent_connection) for accessing content protected by Webthority. Each of these connections can be used by the browser to send multiple HTTP requests to the Webthority proxy. Webthority will close a connection after either processing 100 HTTP requests or after the connection has been idle for 60 seconds, after which the browser will establish a new connection the next time it needs to make an HTTP request. For each concurrent persistent HTTP connection Webthority will use one Tomcat processor. By default 5 processors are available when the Webthority service is started which can support 5 concurrent connections. If more than 5 simultaneous connections are received, then additional processors are created up until the maxthreads value defined in the server.xml file. Additional connections over this maximum value are queued up to 100 connections. Further connection attempts will receive a Connection refused error. The Tomcat processor associated with the connection is released after the connection has been closed and Webthority has finished processing the last HTTP request. If more than 50 Tomcat processors are unused, those above the maximum idle value of 50 will be stopped. 12
Memory For a production environment we recommend that each Webthority host has 4GB of physical memory for both 32bit and 64bit hosts. For 32bit versions of Webthority 512MB of this memory should be allocated to the Java virtual machine heap (an area of memory used by the JVM for dynamic memory allocation http://en.wikipedia.org/wiki/java_virtual_machine#heap). 64bit versions of Webthority should be allocated 2GB to the Java virtual machine heap. For a 32bit process (such as Tomcat.exe used by Webthority) the maximum amount of memory available to it is limited to roughly 2GB. Java applications allocate a block of this memory to store objects known as the Java virtual machine heap or simply, the Java heap, but they also require some memory outside of the Java heap which is why we do not recommend setting the Java heap size beyond 512MB for the 32bit versions of Webthority. 64bit processes do not have this memory limitation so the Java heap size for the 64bit versions of Webthority can be set much higher without starving the Webthority process of memory outside of the Java heap. To configure the maximum amount of memory allocated to the Java heap on Windows, edit <WEBTHORITY_HOME>\bin\svc_in.bat on each Webthority host and add or change the -jvmmx memory option before the --JvmOptions option, making sure you leave a space before and after. For example: 32bit: --JvmMx=512 --JvmOptions="-Xrs; 64bit: --JvmMx=2048 --JvmOptions="-Xrs; 13
On Linux, edit <WEBTHORITY_HOME>/bin/start.sh on each Webthority host and add -Xmx memory option before the -Xrs option, making sure you leave a space before and after. For example: 32bit: $CATALINA_OPTS -Xmx512m -Xrs 64bit: $CATALINA_OPTS -Xmx2048m -Xrs To apply the setting on Windows run <WEBTHORITY_HOME>\bin\svc_out.bat followed by <WEBTHORITY_HOME>\bin\svc_in.bat to update the Windows service and restart Webthority or on Linux run <WEBTHORITY_HOME>/bin/stop.sh followed by <WEBTHORITY_HOME>/bin/start.sh to restart the Webthority service HTTP Connections The number of HTTP connections Webthority can handle concurrently is configurable. For a production environment the recommended default settings described below will allow each 64bit proxy host to handle up to 12,000 persistent HTTP connections concurrently or 3000 connections for 32bit proxy hosts. A further 100 connections will also be queued should this limit be reached before additional connections are refused. Authentication hosts typically only need to support a much smaller number of concurrent HTTP connections and can be left at their default setting allowing 200 concurrent connections. This is because their primary function is to handle the user authentication which typically occurs at the beginning of a user's session. Once a user has been authenticated the proxy host will cache the user's session details to save having to contact the administration and authentication host to validate each request. 14
To increase the number of concurrent HTTP connections, edit the file <WEBTHORITY_HOME>\conf\server.xml on each Webthority proxy host and update the section shown below so that the maxthreads setting is set to either 12000 for 64bit versions of Webthority or 3000 for 32bit versions. For hosts containing a proxy service, the server.xml file will contain multiple instances of this setting, two for each proxy service. One associated with the port number through which users will access the proxy service, e.g 80 or 443, and another for the internal management port which will have a value of 8554 or greater. Only update the maxthreads setting associated with the port number that users will use to access the proxy service, for example port 80 or 443. The other instances can be left at their default value of 200. In situations where a proxy host has multiple proxy services, the value shown below should be split between them so that their combined total is equal to the total number of connections the proxy host can support. This can either be an equal split or weighted in favor of the proxy service which will receive the most connections, e.g 1000 on one and 2000 on the other. The number of connections used by each proxy service can be monitored using the instructions in Monitoring the Number of Concurrent HTTP Connections. 32bit: <Service name="passgo-proxy443"> <Connector protocol="http/1.1" port="443" maxthreads="3000" 64bit: <Service name="passgo-proxy443"> <Connector protocol="http/1.1" port="443" maxthreads="12000" In order to support the higher number of concurrent HTTP connections on a proxy host using a 32bit version of Webthority, you will need to add an additional setting to each proxy host to reduce the amount of memory (addressable memory outside of the Java virtual machine heap) used by each Tomcat processor. 15
To apply this setting on Windows, edit <WEBTHORITY_HOME>\bin\svc_in.bat and place the setting --JvmSs=128 before the --JvmOptions option, making sure you leave a space before and after. For example: --JvmMx=512 --JvmSs=128 --JvmOptions="-Xrs; On Linux hosts you will need to edit <WEBTHORITY_HOME>/bin/start.sh and add the -Xms memory option before the -Xrs option e.g. $CATALINA_OPTS -Xmx512m -Xms128 -Xrs To apply the setting on Windows run <WEBTHORITY_HOME>\bin\svc_out.bat followed by <WEBTHORITY_HOME>\bin\svc_in.bat to update the Windows service and restart Webthority or, on Linux, run <WEBTHORITY_HOME>/bin/stop.sh followed by <WEBTHORITY_HOME>/bin/start.sh to restart the Webthority service. Windows Server 2008 R2 has a constraint which will allow approximately 8000 connections. This is due to Ephemeral port settings (http://en.wikipedia.org/wiki/ephemeral_port). To allow a greater number of connections you will need to run a command in the command prompt to increase the number of dynamic ports e.g. netsh int ipv4 set dynamicport tcp start=40000 num=25000. This example will allow approximately 12000 persistent HTTP connections. This setting will take place immediately and will require a reboot. On Linux you may need to alter your ulimit settings in order to allow Webthority to handle the large number of persistent HTTP connections. In order to do this you can add for example, ulimit -u 25000 and ulimit -n 25000 to the beginning of <WEBTHORITY_HOME>/bin/start.sh and then run <WEBTHORITY_HOME>/bin/stop.sh followed by <WEBTHORITY_HOME>/bin/start.sh to restart the Webthority service. 16
Logging The amount of logging provided by Webthority is configurable. For a production environment the following log levels are recommended to avoid excess logging which can impact performance. To set the logging levels sign-in to the Webthority Administration Console and select each of the following components in the navigation tree, then go to the Logging tab on the right-hand side to access the logging options. Any changes to these settings will take effect dynamically. There is no need to restart the Webthority service. Proxy Service: Authentication Service: Session Manager: Errors, Warnings, Access Errors, Warnings, Authorization Errors, Warnings, Info The Webthority logs directory should be archived at regular intervals to ensure sufficient disk space is available for new logs. 17
Additional Considerations This section contains additional information that you should consider when configuring Webthority. Configuration Validation Once the recommended default production values have been set and before making the environment live, you can validate that the environment can sustain the required number of users using a tool such as JMeter or Grinder to simulate the required level of traffic. SocketBuffer and BufferSize Settings The socketbuffer and buffersize settings improve speed performance for applications which transfer large files. The example below shows how to add these to your proxy settings in the server.xml file: <Service name="passgo-proxy443"> <Connector protocol="http/1.1" /> port="443" maxthreads="200" buffersize="65536" socketbuffer="65536" enablelookups="false" scheme="https" secure="true" SSLEnabled="true" sslprotocol="tls" clientauth="false" keystorefile="j2sdk/jre/lib/security/cacerts" Now restart and resynchronize Webthority for your changes to take effect. 18
Troubleshooting Error: Error allocating socket processor java.lang.outofmemoryerror: unable to create new native thread The out of memory error is referring to addressable memory outside of the Java heap. Each HTTP connection received by Webthority requires a separate thread known as a Tomcat Processor to process each of the HTTP requests received on the connection. Each thread requires a small block of memory outside of the Java heap known as the thread stack in addition to the memory used within the Java heap. To overcome the issue, first ensure that you have correctly added the -Xss128K option, as described in the HTTP Connections section, if using a 32bit version of Webthority to reduce the amount of memory used by each Tomcat Processor. If you still receive the error after adding this setting you will either need to reduce the Java heap size setting -Xmx (thus increasing the memory available for thread stacks) or reduce the maxthreads value and deploy an additional proxy clone. Instructions for changing the Xmx and maxthreads values can be found in the Memory and HTTP Connections sections. Error: Socket accept failed java.lang.outofmemoryerror: Java heap space The out of memory error is referring to available memory within the Java heap. Each HTTP connection received by Webthority requires a separate thread known as a Tomcat Processor to process each of the HTTP requests received on the connection. Each thread requires a small block of memory outside of the Java heap known as the thread stack in addition to the memory used within the Java heap. To overcome the issue you will either need to increase the Java heap size setting -Xmx or reduce the maxthreads value and deploy an additional proxy clone. Instructions on changing the Xmx and maxthreads values can be found in the above Memory and HTTP Connections sections. On a 32bit version of Webthority, take care when increasing the value above 512MB as this will reduce the number of concurrent HTTP connections that Webthority can handle. Monitoring the Number of Concurrent HTTP Connections To monitor the number of concurrent HTTP connections, use the following commands on each proxy host. Replace the IP address and port with that of your proxy service. 19
A space is required after the port number to avoid picking up unwanted TCP connections. List all Current HTTP Connections C:\> netstat -an find "ESTABLISHED" find "192.168.10.109:80 " TCP 192.168.10.109:80 192.168.10.110:3257 ESTABLISHED TCP 192.168.10.109:80 192.168.10.110:3258 ESTABLISHED TCP 192.168.10.109:80 192.168.10.110:3259 ESTABLISHED Count all Current HTTP Connections C:\> netstat -an find "ESTABLISHED" find /c "192.168.10.109:80 " 20
Webthority Backup and Restore Backup This section describes the steps needed to perform a complete backup and restore of Webthority. Webthority Software and Configuration Files The Webthority software and configuration files are stored in a single location specified during the installation procedure. The default locations for Windows and Linux installations are shown below. We recommend that this location is backed up daily and stored in a safe location. The backup of the Webthority files can take place while Webthority is running, however it is recommended that the backup is performed outside of peak hours. Windows C:\Program Files\Quest Software\Webthority Linux /opt/quest/webthority Webthority Datastore Version 6.6 of Webthority includes an internal database used by the Datastore component. The contents of the database are stored within the Webthority installation directory on the Authentication host. To safely backup the database as part of the Webthority installation directory backup, (refer to Webthority Software and Configuration Files), Webthority creates a daily backup of its internal database. This enables a regular filesystem backup of the Webthority installation directory to safely backup an offline version of the database while Webthority is running. The backup time for the internal database can be configured in the Webthority Administration Console under the Datastore component. The time should be configured so that the database backup occurs before the daily filesystem backup of the Webthority installation directory. A one off backup can also be taken using the Backup Now button in the Webthority Administration Console. 21
Start Menu Shortcut Files On Windows, Webthority creates Start Menu shortcut files during installation or upgrade. These are created in the location shown below and should be backed up as part of a full OS backup. Windows 2008 C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Quest Software\Webthority Windows 2003 C:\Documents and Settings\All Users\Start Menu\Programs\Quest Software\Webthority Registry Keys On Windows Webthority stores the location of the Webthority installation files and Start Menu shortcuts in two registry keys in the following location during installation and upgrade. These keys should be backed up as part of a full OS backup. HKEY_LOCAL_MACHINE\SOFTWARE\Quest Software\Webthority 22
Restore On each Webthority host perform the following steps to restore Webthority from backup. If you are recovering from an OS failure, first restore the full OS backup using the last known good backup. 1. Stop the Webthority service and, on Windows, remove its Windows Service entry. Windows C:\Program Files\Quest Software\Webthority\bin\svc_out.bat Linux /opt/quest/webthority/bin/stop.sh On hosts running Microsoft Windows the svc_out.bat and the later svc_in.bat in step 4 should be run from a command prompt rather than double clicking the files in Windows Explorer. If the host has User Account Control enabled then the command prompt must be started using the "Run as Administrator" option. 2. Restore the Webthority installation directory with the last known good backup. This can be either a backup of the same Webthority version or an earlier version taken before upgrade. The default location for Windows and Linux is shown below. Windows C:\Program Files\Quest Software\Webthority Linux /opt/quest/webthority 23
3. If reverting to a previous version of Webthority on Windows you may also need to restore the Start Menu shortcuts as they may have changed. To do this restore the files from the following location. Windows 2008 C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Quest Software\Webthority Windows 2003 C:\Documents and Settings\All Users\Start Menu\Programs\Quest Software\Webthority 4. Start the Webthority service and, on Windows, create its Windows Service entry. Windows C:\Program Files\Quest Software\Webthority\bin\svc_in.bat Linux /opt/quest/webthority/bin/start.sh 5. Sign in to the Webthority Administration Console. 6. To synchronize the Webthority hosts, right-click each host and then click ReSync. 7. Finally, restore the internal database used by the Datastore component. To do this, expand the Authentication host and click the Datastore component. Select the Backup tab and click Restore. 24