Tuning Guide for BEA WebLogic Server on HP NonStop Servers Author: Santosh Menon November 17, 2005

Transcription

1 Tuning Guide for BEA WebLogic Server on HP NonStop Servers Author: Santosh Menon November 17, 2005

2 TABLE OF CONTENTS 1.0 Introduction Background Information Connection Management The Need for Connection Pooling Connection Pooling as Described in the JDBC Specification WebLogic Server Connection Pooling JDBC Statement Pooling WebLogic Server Execute Queues WebLogic Server Socket Handling Configuration Guidelines Configuring WebLogic Server on HP NonStop Server for Accessing SQL/MX Configuring a Connection Pool with the non-xa Driver Configuring a Connection Pool with the XA Driver Configuring a Non-Transactional Datasource Configuring a Transactional Datasource Disabling the WebLogic Server Prepared Statement Cache Configuring the Number of Connections in a Connection Pool Configuring WebLogic Server Execute Threads Configuring Socket Reader Parameters Configuring TCP/IPV6-related Parameters Setting the TCP/IP Process Name Configuring the Listen Address for WebLogic Server Memory Considerations: Moving QIO to KSEG Restricting the List of Ports that Can Be Filtered Ephemeral Ports Configuration Reducing the Minimum Listen Queue Size and Acknowledgement Delay Interval Logical Network Partitioning Fault Tolerance Considerations Performance Considerations Number of WebLogic Server Instances per CPU SQL/MX Nowait Considerations Load Balancing Disabling the Application Poller Java Heap Setting Using the WebLogic Server Console to Monitor the Performance of an Application Tuning the Java Heap JMX-Based Custom Monitoring Additional Tuning Considerations Improving Instance Startup Time JMS Considerations Configuring TMF and DP2 for Use with WLS Tuning the SQL/MX Database and the OSS File System Tuning TLOG Files Tuning the Data Cache...28

3 1.0 Introduction The purpose of this paper is to discuss configuration guidelines, development best practices, and tuning information for BEA WebLogic Server for HP NonStop servers. This document builds on experience configuring clusters with up to 60 instances running on four 16-CPU HP NonStop servers, as well as on documents 1, white papers, FAQs available from BEA on WebLogic Server, and information from several technical resources in BEA and HP NonStop Professional Services who provided valuable insight into the product. This document is composed of the following topics: Information about several WebLogic Server features, as background for configuration guidelines NonStop server-specific configuration guidelines for Connection pooling, Statement caching, and management of Thread Count and the XA Resource Manager Fault tolerance and performance considerations Discussions in this paper are based on WebLogic Server 8.1 SP3 as deployed on a NonStop server, though externals for later versions (and possible changes that can affect NonStop server deployments) are provided in some instances. Information on NonStop server functionality refers to functionality in the G06.20 or later operating system versions, which are certified by BEA for use with BEA WebLogic Server. 2.0 Background Information WebLogic Server is an application server a platform for developing and deploying multi-tier distributed enterprise applications. WebLogic Server centralizes application services such as Web server functionality, business components, and access to backend enterprise systems. It also provides enterprise-level security and administration facilities. 2 WebLogic Server implements J2EE, the Java Enterprise standard, which includes the Java programming language as well as J2EE component technologies for developing distributed objects. The WebLogic Server architecture consists of three tiers: A client tier, which contains programs executed by users such as web browsers and network-capable application programs. A middle tier, with servers addressed directly by clients such as WebLogic Server, other web servers, firewalls, and proxy servers that mediate traffic between clients and WebLogic Server. A backend tier, with enterprise resources such as database management systems (DBMS), mainframe applications, transaction monitors, and packaged enterprise resource planning (ERP) applications. The rest of this section provides background material for configuring BEA WebLogic Server on HP NonStop servers. 1 is an excellent resource on WebLogic Server. The FAQs are located at 2 Information in these paragraphs is from BEA product information. For further information about BEA WebLogic Server architecture, see 3

4 2.1 Connection Management A key area of resource usage in multi-tier enterprise applications is the definition and management of connections between applications and databases. One simple type of connection is a one-to-one mapping between an application and a database. An application requests a database connection from the relevant database driver and executes various statements using the connection. When finished, the application releases the connection. The following code fragment shows how a typical standalone application might acquire and use JDBC Connections using the DriverManager interfaces. Java applications (in particular J2EE applications) can also use the Datasource interfaces to obtain connections. The purpose of the code below is to introduce the user to the management of connections in application programs prior to the introduction of connection pooling concepts. An application using the DriverManager interface is provided below, followed by one using the DataSource interface. Code Listing 1: Sample JDBC application using DriverManager interface try { String dbdriver = com.dbvendor.driver ; String dburl = jdbc.vendordb.specific-optional-string ; String username = scott ; String password = tiger ; // The above values can also be looked up from a JNDI tree // dynamically to avoid hardcoding them in the source code Class.forName(dbdriver).newInstance(); Connection dbconn = DriverManager.getConnection(dburl, username, password); // Execute db statements using the connection dbconn.close(); } catch (Exception e) { //handle exceptions } Code Listing 2: Sample JDBC application using DataSource interface // As part of application configuration/setup, a Datasource object // for a database is bound to the JNDI. Now if the database used // changes from one vendor to another, only the JNDI registration // part of application environment has to change. VendorDataSource v = new VendorDataSource(); try { Context ctx = new InitialContext(); ctx.bind( jdbc/warehouseds, v) } catch (NamingException exc) { // handle exceptions } 4

5 // // Once the DataSource is registered in the JNDI namespace an // application can lookup the Datasource and use it // try { Context ctx = new InitialContext(); DataSource ds = (DataSource) ctx.lookup( jdbc/warehouseds ); Connection dbconn = ds.getconnection(username, password); // Execute db statements using the connection dbconn.close(); } catch (Exception e) { //handle exceptions } In the preceding code fragments, the Connection object represents a physical connection to the database. There is a one-to-one mapping between the client connection and the physical connection to the database. 2.2 The Need for Connection Pooling When many application processes operate in parallel, the one-to-one association between logical and physical connections can be very expensive due to the significant overhead in establishing each connection. In a J2EE application server environment, for example, applications share resources so they can be equitably distributed and efficiently used. The common model in J2EE environments is for application components to own a resource only for the duration of the work performed by the resource. Applications are expected to follow an acquire use- release sequence when using resources. If J2EE components such as EJBs and servlets were to follow the one-to-one connection handling model in the preceding code example, each request would incur a significant overhead primarily to create the connection and initialize the session with the database server. Moreover, in a J2EE environment, as in any application framework, the application server needs to manage connections to make sure resources are efficiently used by all deployed applications. Use of a one-to-one connection model would restrict that ability. A more efficient alternative is to share a connection across many processes. To support this, the standard extension specification for the JDBC 2.0 specification introduced a mechanism called connection pooling. A JDBC connection pool represents a pool of connections to the respective database and is configured with minimum and maximum capacities. Once a database connection is opened, the connection pooling mechanism lets these database connections be shared by all applications that need database access. The overhead associated with establishing a connection is incurred only once for each connection in the pool, instead of once per client request. The J2EE application server monitors the pool of database connections, refreshing them as needed and ensuring reliable database services for applications. 2.3 Connection Pooling as Described in the JDBC Specification In the JDBC Connection pooling model, the database driver provides a ConnectionPoolDataSource implementation and the application server provides the connection pooling infrastructure using the facilities provided by the JDBC driver. The sequence of events shown below describes the steps that occur in preparation for JDBC connection pooling for a DataSource. 5

6 1. The application server is configured with information about the JDBC driver ConnectionPoolDataSource. 2. On startup, the application server pre-creates the DataSource for each configured database in the application server. 3. The server gets a set of PooledConnection objects from the ConnectionPoolDataSource and caches them. Each PooledConnection object represents a physical database connection. This is done as per the configuration for each DataSource. 4. The server registers itself as a listener for the connection events on these PooledConnection objects. This alerts the JDBC driver to notify the application server when the connection is closed or an unrecoverable error occurs on the connection. 5. Whenever the server needs to provide a JDBC connection to an application component, it obtains a logical connection object from the PooledConnection object and provides it to the application. The sequence of events shown below describes the Connection pooling activity when a client acquires a connection and closes the connection. 1. When the JDBC application needs to access the database, it issues a getconnection request using the DataSource object that the application server created in Step 2, above. 2. Upon this request, the application server looks at its list of PooledConnection objects that are not in use and chooses one of them. 3. The application server gets the logical Connection object associated with the PooledConnection from the JDBC driver. 4. The logical Connection object is returned to the application. 5. The application uses the logical Connection object to perform database (DB) work. 6. When the application is done using the (logical) connection object, it closes the connection. This close request triggers the database driver to reset the PooledConnection so that it can be reused. The JDBC driver does not do a physical close of the DB connection, but cleans up the Connection for later reuse. 7. The JDBC driver notifies all listener objects registered on the PooledConnection object about the close request so they can take any necessary action. 8. An application server implementing the infrastructure for connection pooling might then mark the state of the PooledConnection as available so the object is free for potential reuse later. In this manner, multiple logical Connections can be created for a given PooledConnection object during the lifetime of PooledConnection. At any time, only the most recent logical Connection object obtained from a PooledConnection object can be used to do database work 3. The application server and the JDBC driver manage the PooledConnection objects and the logical Connection lifetimes. The following diagram illustrates the overall process: 3 It is legal for a JDBC connection pooling consumer to get the logical Connection from a Pooled Connection multiple times without an intervening close on the logical connection. The JDBC Specification makes it clear that when a logical connection is acquired from a pooled connection, the prior open logical connection is automatically closed (without notifications to listeners). 6

7 1 getconnection JDBC Application 4 Return the logical connection to caller Application uses the 5 connection to do DB work Choose a free Pooled Connection (create if necessary) 2 Application Server Pooled Connection Object Cache 8 Mark PooledConnection as free for later reuse 6 Application closes the connection 3 Get a logical connection from the pooled Connection 7 Driver notifies application server so that the Pooled Connection can be reused for future connection requests JDBC Driver When the application server uses JDBC driver connection pooling facilities, it transparently uses those services in response to application requests. The JDBC Connection pooling infrastructure avoids costly connection creation by letting connections be requested and used by multiple applications. Note: XAConnections work like PooledConnections. XAConnection is derived from the PooledConnection and thus it supports the event listening model applicable to the PooledConnection objects. In addition, XAConnection objects also provide the facilities for a Connection to participate in global 4 transactions. 2.4 WebLogic Server Connection Pooling WebLogic Server implements Connection pooling differently than the JDBC model. Instead of relying on the JDBC driver to provide connection events, it uses wrapper objects for each JDBC driver object. These wrapper objects intercept method calls on the corresponding JDBC objects and take action as needed. As shown above, in the JDBC connection model the application server caches the PooledConnection object and gets a new logical Connection from the JDBC driver every time the application requests a Connection. WebLogic Server, on the other hand, does not depend on notifications from the JDBC driver to manage connections. When a Connection pool is created, based on the configuration for the connection pool, WebLogic Server acquires the database connections from the driver and caches those connections. These connections are physical DB connections when acquired from the JDBC driver implementing the java.sql.driver interface. 4 A global transaction (also called a distributed transaction) has one or more transaction branches. Each branch represents a portion of the work in support of a global transaction for which a transaction manager (TM) and set of participating resource managers (RMs) engage. 7

8 When XADataSources are used, WebLogic Server gets the XAConnection objects from the JDBC driver and acquires the associated logical Connection object from the XAConnection and caches them both. The WebLogic Server needs the XAConnection object to perform transactional work using the XA facilities and it uses the logical Connection object to wrap and pass onto the application on demand. The following diagram outlines the WebLogic Server Connection pooling Model. The same model is used for XAConnections WebLogic Server obtains the logical Connection from the JDBC driver once and uses it until it closes the XAConnection object. getconnection Choose a free Connection object or create one (if necessary) Wrap the object within the WebLogic Server Connection object JDBC Application 4 Return the WebLogic Server wrapper object 5 to the caller Application uses the connection for database work delegated to the JDBC driver by the wrapper WebLogic Server Connection Object Cache JDBC Driver 6 7 Application closes the connection 8 Wrapper, using JDBC APIs, cleans up the Connection object and marks it for reuse Wrapper delegates database work to the real DB connection WebLogic Server connection pooling complements the JDBC driver connection pooling. When the JDBC driver provides Connection pooling for its connections, the JDBC driver pooling mechanism is used only when the WebLogic Server Connection pooling implementation closes a connection. When the WebLogic Server connection pooling infrastructure closes the Connection object, it removes it from its Cache. At that point, if the JDBC driver has connection pooling enabled, the JDBC driver, instead of physically closing the connection, keeps the connection available for potential later requests from WebLogic Server. 8

9 2.5 JDBC Statement Pooling JDBC 3.0 introduced the concept of statement pooling, where a JDBC Statement object is tightly coupled with the underlying physical DB Connection. A statement prepared for one connection cannot be used with another connection. Note that statement pooling applies only to PreparedStatement objects. It is recommended that all applications use PreparedStatements to prepare and execute SQL statements. JDBC drivers implement statement pooling as follows: 1. The driver intercepts the close method call on the PreparedStatement object. 2. Instead of performing a physical close, the driver moves the object to a cache (after performing any necessary cleanup on the PreparedStatement object so that it can be reused). The PreparedStatement object is typically identified in the cache by the SQL statement, isolation level of the transaction, and other relevant information. 3. When a request to prepare the same SQL statement (with the same execution settings) is made, the JDBC driver uses the cached object instead of going through the process of requesting the Database server to prepare the statement again. This provides an opportunity for the JDBC driver to pool PreparedStatement objects such that a given SQL statement is prepared only once for a DB connection regardless of the number of logical Connections created off of it. Statement pooling makes sense only with connection pooling, where logical Connection objects are closed but the physical DB connection is long-lived. Even though Statement pooling is formally introduced only in the JDBC 3.0 specification, many JDBC vendors have implemented statement pooling functionality in their existing JDBC 2.x compliant drivers (which is the specification mandated by J2EE 1.3). The WebLogic Server implements statement pooling differently. Its implementation is independent of JDBC driver pooling functionality. Statement pooling is referred to as Statement Caching in WebLogic Server. As with Connection pooling, WebLogic Server creates a wrapper for all JDBC objects. When WebLogic Server Statement caching is enabled, these wrapper objects intercept the close request and do the necessary work to clean up the PreparedStatement object. Instead of delegating the close request to the underlying JDBC object, the wrappers clean up the Statement object (using JDBC APIs) and move the statement to a free pool maintained by WebLogic Server for later reuse. 2.6 WebLogic Server Execute Queues In WebLogic Server, one of the critical resources configured for each managed server instance is the execute queue. An execute queue represents the configuration of a work item queue, an associated pool of threads servicing the queue and the thread priorities. The actual usage of a thread pool is dependent on its execute queue. There are many predefined WebLogic Server system execute queues, and there can also be application defined execute queues. Execute Queues form the basis for the staged processing architecture of WebLogic Server. One of the main execute queues in WebLogic Server is named the kernel queue (or the default queue). This execute queue specifies the thread pool for the execution of WebLogic Server internal requests, application requests, and handling sockets, among other things. An application can optionally define its own execute queue for handling the application requests. If such a queue is not defined for an application, requests for that application component are handled using the default execute queue. 9

10 When using the Java socket multiplexer (identified by Dweblogic.nativeIOEnabled = false), the default queue also determines the number of threads that can do socket reading work for a WebLogic Server instance, which is specified as a percentage of the total number of threads configured in the default execute queue. The number of socket readers configured, the multiplexer mechanism, and the expected number of socket connections to a WebLogic Server has important bearing on the default execute queue thread count. WebLogic Server socket handling is discussed below. When the native socket multiplexer is used (identified by Dweblogic.nativeIOEnabled=true, which is the default for WebLogic Server 8.1 on an HP NonStop server), the socket readers are configured from a separate execute queue. Moreover, the number of socket readers is very small (by default, 3). These socket readers use operating system facilities (select() in most Unix systems and an equivalent thread-aware select() function on the NonStop server), which allows the threads to wait for an IO event instead of polling them. This provides significant savings by reducing the CPU usage, reducing the number of threads in the default execute queue, and simplifying the configuration of the default execute queues. WebLogic Server also has an experimental NIO multiplexer (that is not supported). NIO is the J2SE package for new IO handling capabilities in the Java Platform. One of the new capabilities is the ability to wait for IO events on multiple sockets channels. 2.7 WebLogic Server Socket Handling WebLogic Server uses a socket multiplexer architecture whereby socket requests are read by a set of threads and forwarded to other execute threads for further processing (thus tying into the staged processing architecture of WebLogic Server). The socket multiplexer architecture lets any of the socket reader threads multiplex between many sockets to check for the readiness of the sockets and read request data off them. WebLogic Server provides native code socket multiplexers for many of the operating system platforms that WebLogic Server runs on. This socket multiplexer uses system provided wait primitives (like the poll, select or other system calls) to wait for ready events on sockets and process them. This also means that only one thread needs to wait for sockets. For those platforms where the native socket multiplexer is not available, WebLogic Server uses a multiplexer written in pure Java. While the Java 1.4 platform (used by WebLogic Server 8.1) provides support for non-blocking operations in the NIO package, a socket multiplexer using this facility is not supported by WebLogic Server, due to lack of functionality and performance issues. To work with the platforms without a native socket multiplexer, WebLogic Server uses the socket timeout facility. A pool of socket readers periodically polls all available sockets for data. Instead of waiting indefinitely, a timeout is set on the socket so that if the socket is not ready with data within the specified timeout, the socket reader thread moves on to poll the next available socket. This mechanism is less efficient than the native polling mechanism. 3.0 Configuration Guidelines 3.1 Configuring WebLogic Server on HP NonStop Server for Accessing SQL/MX As shown earlier, to access a database an application looks up a Datasource object configured in a WebLogic Server instance (using JNDI) and acquires JDBC connections from the Datasource object. WebLogic Server JDBC Datasource objects, which can be transactional or non-transactional, act as the interface between applications and the JDBC connection pool. The Datasources use connection pools configured within WebLogic Server for acquiring connections. When a Connection pool is configured in WebLogic Server, the JDBC driver providing the connections for the connection 10

11 pool can be an instance of class java.sql.driver (for JDBC/MX, this is the com.tandem.sqlmx.sqlmxdriver) or an instance of javax.sql.xadatasource (for example, com.hp.nsk.xares.wlstxsqlmxdatasource). The former is hereafter referred to as the non-xa driver and the latter as the XA driver. Which driver to choose depends primarily on whether the associated WebLogic Server DataSource will participate in global transactions involving other XA resources 5 : If the Datasource will not participate in a transaction involving other XA aware resources, specify the non- XA driver for the connection pool. Otherwise, specify the XA Driver. When configuring connection pools with the non-xa driver, WebLogic Server uses the JDBC driver, which is an instance of java.sql.driver (non-xa driver), for creating its Connection Pool. When configuring connection pools with the XA driver, WebLogic Server uses the XA driver (which is an instance of javax.sql.xadatasource). From the application administrator perspective, connection pooling for JDBC resources is achieved by configuring the corresponding JDBCConnectionPool element in the WebLogic Server configuration file. The JDBC driver connection pooling parameters should be set so they don t interfere with the WebLogic Server connection pooling mechanism. Examples of WebLogic Server connection pooling configuration are provided below. 3.2 Configuring a Connection Pool with the non-xa Driver The JDBC/MX product provides non-xa drivers for accessing NonStop SQL/MX databases. There are implementations of java.sql.driver (which is what we will use with WebLogic Server) and also implementations of the java.sql.datasource. Version 3.0 of the JDBC/MX product, which works with the NonStop SQL/MX release, supports both Connection pooling and Statement pooling, conformant with the JDBC specification. Similarly, Version 3.1 of the JDBC/MX product works with the SQL/MX 2.0 release providing the same functionality. Additionally, the JDBC/MX 3.1 version also supports LOB data-types as an emulated datatype with the JDBC driver. One of the components of the JDBC/MX product is the driver object implementing the java.sql.driver interface. The class name of this driver object is com.tandem.sqlmx.sqlmxdriver. When a Connection Pool is created with the JDBC/MX non-xa java.sql.driver object as its underlying driver, WebLogic Server understands that the JDBC Connections acquired from the Connection pool are not XA aware and cannot typically take part in global transactions involving multiple resources. When configuring the WebLogic Server Connection Pool, it is recommended that Statement pooling in the JDBC driver be enabled by specifying the maxstatements property as shown below. When configuring WebLogic Connection Pools, JDBC driver-specific properties are passed to the driver using the Properties element of the JDBCConnectionPool configuration element 6. Properties= maxstatements=<number> Where <number> is the maximum number of statements in the pool. An example extract from the WebLogic Server configuration is provided in the following sections. 5 A connection pool configured with the XA driver can also be used by non-transactional datasources. 6 While the configuration for WebLogic Server-configurable components are described in terms of the configuration file (config.xml), it is recommended that the WebLogic Server Administrator s console be used. 11

12 3.3 Configuring a Connection Pool with the XA Driver The alternative to creating a connection pool with the non-xa JDBC/MX Driver name is to create it with the XAaware driver. The Connection pool can then be used to create transactional Datasources that can be used in global transactions involving multiple resources such as JMS. The XADataSource name for SQL/MX (instance of javax.sql.xadatasource) is com.hp.nsk.xares.wlstxsqlmxdatasource. The JDBC/MX drivers that are part of the JDBC/MX product do not support the XAResource interface because XA is not used in the NonStop server for coordination of NonStop server Resources. However, WebLogic Server expects a JDBC driver to support the XADataSource interface to fully participate in global transactions, or when a Datasource is configured for container managed services (like CMP). So, the XA-aware driver for SQL/MX is provided with the HP NonStop Server Toolkit for BEA WebLogic Server 8.1 SP3 7. As with the non-xa Driver, it is recommended to enable statement pooling for the Connections by specifying the system property maxstatements or configuring the Connection pool with the property maxstatements. 3.4 Configuring a Non-Transactional Datasource To configure a non-transactional Datasource in WebLogic Server, the underlying connection pool can be configured with the XA-aware driver or the non-xa aware driver. It is recommended that the non-transactional DataSource for SQL/MX be configured with the non-xa driver instead of the JDBC/MX XA-aware driver. Note that as far as WebLogic Server is concerned, the connections acquired from this Datasource do not take part in the WebLogic Server global transactions. However, on HP NonStop servers, a static XA resource manager is configured in each instance of WebLogic Server. This static XA resource manager is enlisted in all WebLogic Server transactions, providing the TMF transaction context. So, whenever a WebLogic Server transaction is started (either explicitly or using declarative transactions), a corresponding TMF transaction with the same scope as the WebLogic Server global transaction is started. This TMF transaction provides the pervasive transaction context for all NonStop resources accessed within the context of the WebLogic Server transaction. So, even though the WebLogic Server transaction manager is not aware of it, the WebLogic Server Transaction is also indirectly providing the scope for the work done through non-transactional Datasources. Depending on the number of resources participating in a transaction, WebLogic Server Transaction Manager can perform several optimizations for efficient transaction management 8. Servlets, Session Beans, etc. can use the non-transaction DataSource with TMF protection achieved through the static resource manager. If the only resource accessed is SQL/MX, the one-phase commit optimization of the WebLogic Server TM can provide significant performance benefits. 3.5 Configuring a Transactional Datasource To configure a non-transactional Datasource in WebLogic Server, the underlying connection pool can be configured with the XA-aware driver or the non-xa aware driver. 7 For clarity, we hereafter refer to this as WebLogic Server NonStop Server Toolkit. 8 These optimizations include one-phase commit optimizations and read-only optimizations where the WebLogic Server Transaction Manager can do away with writing log records if all or all but one participating resources do not report modified state. 12

13 WebLogic Server will involve the transactional Datasources in its distributed transactions 9. To do this, the underlying JDBC resource should be XA aware. When a transactional Datasource is configured based on a non-xa aware driver, WebLogic Server will provide the XA emulation. It is not generally advisable to use this because: Only one non-xa aware Datasource (with XA emulation from WebLogic) can participate in a global transaction. The XA-aware driver com.hp.nsk.xares.sqlmxdatasource driver takes precautions to ensure that a proper transaction context is always available for processing JDBC connections. When the transactional Datasource is configured with the XA-aware SQL/MX driver (which is a wrapper provided with WebLogic NonStop Server Toolkit), the Datasource can be used by all EJB and web components that access multiple resources. The XA resource that is part of the JDBC/MX XA-aware driver depends on the static XA resource to perform transaction branch coordination between the WebLogic Server TM and TMF. The XA resource in the driver responds with a READONLY answer to prepare requests, which lets the WebLoic Server TM perform optimizations with respect to writing logs. 3.6 Disabling the WebLogic Server Prepared Statement Cache Performance tests have shown that using WebLogic Server Prepared Statement Cache has more overhead in terms of the path-length on NonStop servers. Disabling the Prepared Statement Cache (the default is that the statement caching is enabled in WebLogic Server 8.1) provides optimal performance when coupled with JDBC/MX statement pooling. To disable the WebLogic Server Prepared Statement caching, specify the attribute StatementCacheSize with a value of 0 in the respective Connection pool configurations. 9 As mentioned before, distributed transactions are also called as global transactions. The term distributed here means that branches are contracted to participating RMs. 13

14 The following is a simple matrix showing the various combinations of connection pooling: WebLogic Server Connection Pool and DataSource Types Connection Pooling Statement Pooling Usage Considerations Non-XA connection Pool with non-transaction Datasource XA connection pool with transactional Datasource Use WebLogic Server connection pooling. Use default JDBC connection pooling. Use WebLogic Server connection pooling. Use default JDBC connection pooling. Disable WebLogic Server Statement caching. Enable JDBC driver statement pooling. Disable WebLogic Server Statement caching. Enable JDBC driver statement pooling. Use to access NonStop Resources from servlets, stateless session beans, etc. when the WebLogic Server transaction is used to access NonStop server resources only. For CMP beans, this cannot be used. When a WebLogic Server transaction accesses NonStop server resources and other resources (like WebLogic Server JMS). As the persistence Datasource for CMP beans For Bean Managed Entity Beans which can be invoked remotely. 3.7 Configuring the Number of Connections in a Connection Pool When configuring Connection Pools, it is recommended that you set the Initial and Maximum Capacity to the same value. This means that all required JDBC Connections will be acquired during application server startup. To specify the initial and maximum capacity of a Connection pool, specify the properties InitialCapacity and MaxCapacity in the JDBC Connection Pool configuration. Performance tests have shown that restricting the total number of connections configured for a connection pool to the absolute minimum required value (typically four connections per connection pool for heavy usage scenarios) provides higher performance 10. Because restricting the number of connections has a direct impact on the number of EJB or servlets that can access the database, it is important that the number of the components accessing the connection pool also be throttled. To throttle the maximum number of EJBs, specify the WebLogic Server deployment descriptor parameter <max-beans-in-free-pool> in the file weblogic-ejb-jar.xml to restrict the maximum number of EJBs that are created by WebLogic Server. It is recommended that this value be set to the maximum connection pool capacity (which ideally is four, as mentioned above for heavy SQL usage scenarios). Examples Sample JDBC Connection Pool Element extracted from a configuration using the non-xa driver: <JDBCConnectionPool CapacityIncrement="1" ConnLeakProfilingEnabled="false" DriverName="com.tandem.sqlmx.SQLMXDriver" InitialCapacity="4" 10 This is a well-known fact that can be applied to many resources use what you need and no more. However, typically in transaction processing environments, there might be an urge to over configure resources. 14

15 MaxCapacity="4" Name="sqlmxPool StatementCacheSize="0" Properties="maxStatements=100" ShrinkPeriodMinutes="30" ShrinkingEnabled="false" SqlStmtMaxParamLength="10" SqlStmtParamLoggingEnabled="false" SqlStmtProfilingEnabled="false" SupportsLocalTransaction="true" Targets="joecluster" TestConnectionsOnRelease="false" TestConnectionsOnReserve="false" URL="jdbc:sqlmx:"/> Sample JDBC Connection Pool Element extracted from a configuration using the XA driver: <JDBCConnectionPool CapacityIncrement="1" ConnLeakProfilingEnabled="false" DriverName="com.hp.nsk.xares.WLSTxSQLMXDataSource" InitialCapacity="4" KeepLogicalConnOpenOnRelease="false" KeepXAConnTillTxComplete="true" MaxCapacity="4" Name="sqlmxPool" NeedTxCtxOnClose="false" NewXAConnForCommit="false" PrepStmtCacheProfilingEnabled="false" PrepStmtCacheProfilingThreshold="10" StatementCacheSize="0" Properties="maxStatements=100" RecoverOnlyOnce="false" RefreshMinutes="0" ShrinkPeriodMinutes="30" ShrinkingEnabled="true" SqlStmtMaxParamLength="10" SqlStmtParamLoggingEnabled="false" SqlStmtProfilingEnabled="false" SupportsLocalTransaction="true" Targets="joecluster" TestConnectionsOnRelease="false" TestConnectionsOnReserve="false" URL="jdbc:sqlmx:"/> 3.9 Configuring WebLogic Server Execute Threads The configuration of Execute Threads is a key aspect of WebLogic Server tuning. However, unlike the 7.0 SP1 release of WebLogic Server on the NonStop server, the 8.1 SP3 release of WebLogic Server on the NonStop server does have native code multiplexer support. This greatly reduces the complexity of configuring the execute threads. By default, WebLogic Server 8.1 SP3 is configured with 25 threads for the production environment and 15 threads for development application environments. Also, it is possible for applications to specify their own execute queue with a dedicated set of threads. 15

16 While it is not recommended, the number of native socket readers can also be throttled. Specify the system property Dweblogic.PosixSocketReaders=<n> where <n> is the number of socket readers desired. The default is three (3), which is ideal for most configurations. While the Java multiplexer is not recommended in the 8.1 SP3 release (and will perform poorly), it can be used. To use the Java socket multiplexer, explicitly disable the native socket multiplexer (by setting the system property - Dweblogic.NativeIOEnabled =false). When the Java socket multiplexer is used, use the following scheme to calculate the Execute Queue Thread Count for each server taking part in a cluster configuration: 1. Determine the number of servers in the cluster. 2. Determine the average number of clients that will connect to the cluster. 3. If session replication is configured (for HTTP sessions and Stateful session beans), add the values obtained in Steps 1 and 2. Otherwise, carry forward the value in Step Add 2 to the value obtained in Step 3. This is the average number of sockets that will be open on the server at any time. 5. Calculate the minimum of the value obtained from Step 4 and add 50. This is the number of socket readers to configure Calculate the number of worker threads. This is the maximum of the value in Step 2 and 25 (the recommended minimum number of worker threads is 25). 7. Add the results of Steps 5 and 6, plus the overhead of 5 execute threads. That is the execute thread count. 8. Calculate (Step 5 * 100) / Step 7. This is the percentage of socket readers. For example, assume a cluster with four servers and an average of 50 client connections with no session replication configured: Step 1: 4 Step 2: 50 Step 3: 50 Step 4: 50+2 = 52 Step 5: Min(50,52) = 50 Step 6: maximum (50, 25) = 50 Step 7: = 105 Step 7: (50*100) / 105 = 48 (rounded up). This example requires 105 threads with 48 percent socket readers. This also means to allow a maximum of 103 execute threads (discounting two for the admin server connection and multicast handling) executing simultaneously in the server. WebLogic Server also provides monitoring of all the active queues (including the default execute queue) using the WebLogic Server console. The Server->Monitor->All Active Queues->(Specific queue, particularly the default queue), Sockets and Connections can be used to identify the number of socket reader threads, the amount of work through them, and the number of worker threads and idle threads in a queue. The throughput information for each thread is also provided Configuring Socket Reader Parameters The WebLogic Server 8.1 SP3 for NonStop server has native code for the socket multiplexer, which means the WebLogic Server will not use its timeout-based polling mechanism as described in the overview section. If native multiplexer is disabled for some reason, WebLogic Server uses a scheme similar to the following for arriving at the polling timeout for each socket (specified in milliseconds). 11 While we have specified a maximum value of 50 for the number of socket readers, it is recommended that the number of socket readers be determined by the target application. Note that increasing the number of socket readers to match the number of open sockets will start providing lower throughput while extracting higher processor time consumption. A value of 50 seems to be a logical starting point, but this value has to be individually ascertained for each application. 16

17 if (numreaders > numsockets) timeout = SocketReaderTimeoutMaxMillis else Timeout = numreaders * / numsockets if (timeout < SocketReaderTimeoutMinMillis) timeout = SockerReaderTimeoutMinMillis if (timeout > SocketReaderTimeoutMaxMillis) timeout = SockerReaderTimeoutMaxMillis SocketReaderTimeoutMinMillis and SocketReaderTimeoutMaxMillis are by default set to 100 and 500, respectively, but can be overridden in the Server element. The attributes of the Server object, SocketReaderTimeoutMinMillis and SockerReaderTimeoutMaxMillis can be used to control the polling timeout in the server. If there are enough socket readers in the Execute queue such that the number of socket readers is greater than or equal to the average number of open sockets on the server, set the SocketReaderTimeoutMaxMillis to a high value (for example, 2 or 3 seconds). This helps reduce the number of polling activities on a socket before a request is read. If the number of socket readers is less than the number of active sockets, WebLogic Server writes a warning message in the server log file. When the number of socket readers is less than the average number of socket readers, the SocketReaderTimeoutMaxMillis can be set to the default value of 500 milliseconds. Note that when the number of socket readers is less than the number of open sockets, an alert is written to the WebLogic Server log file. This log file can be periodically monitored and, if needed, the number of socket readers configured can be increased. For example, consider a WebLogic Server instance with 10 socket reader threads and 30 active sockets. In this scenario: numreaders = 10 numsockets = 30 timeout = 10 * 60000/30 = 10 * 2000 = milliseconds. Since the timeout value (20000) is greater than the default maximum (which is 500), the default maximum is used Configuring TCP/IPV6-related Parameters WebLogic Server on Guardian requires TCP/IPV6 stack configured in INET4 mode. This section describes configuration options needed for TCP/IPV Setting the TCP/IP Process Name TCP/IPV6 cannot be configured as the default TCP/IP process on a NonStop server. The Guardian define name =TCPIP^PROCESS^NAME can be used to make TCP/IPV6 stack the default stack of an application. The following command sets the define =TCPIP^PROCESS^NAME to a TCP/IP provider named $ZSM1: $ add_define =TCPIP^PROCESS^NAME class=map file=\$zsm1 Once TCP/IPV6 is specified as the TCP stack used by the WebLogic Server process, an additional define must be specified (=PTCPIP^FILTER^KEY) so that multiple WebLogic Server processes can join a multicast socket group. 17

18 All WebLogic Server instances of one WebLogic Server cluster (and only those instances) should use the same key. It can be set to the WebLogic Server cluster name itself, but is subject to Guardian filename rules (8-character maximum length, with the first character an alphabetical character). The following command sets the filter key to cluster1: $ add_define =PTCPIP^FILTER^KEY class=map file=cluster1 When nodemanager is used, these defines need to specified in the shell script that starts the nodemanager Configuring the Listen Address for WebLogic Server The WebLogic Server stores the ClusterAddress and expects to reach a surviving member of a cluster for implementing failover for Entity beans and EJB Handles. Therefore, the following guidelines are recommended: Configure each instance of the WebLogic Server in a cluster to listen on different IP addresses but use a common port. This is the recommended configuration. According to the WebLogic Server documentation, failure to adhere to this recommendation means that entity bean failover and EJB handle failover will not work. Moreover, the non-dns form of cluster addresses (where the cluster address is specified as a list of IP:port values) is recommended only for development use by BEA. Set up a DNS name that maps to all the IP addresses used by instances that are part of a WebLogic Server cluster. This DNS name should not map to any other IP address not used by the cluster members. For example, if a cluster is set up with four member instances: Configure a single common port as the listen port for all the 4 servers in the cluster. Configure each server instance with its own IP address. Set up a DNS name that maps to these 4 IP addresses only. Note: this DNS name is specified in the WebLogic Server configuration as the ClusterAddress for a particular Cluster. While it might look daunting that large clusters require many more addresses if we go with one IP address per instance, the mitigating factor is that the TCP/IP V6 stack lets Alias IP addresses be specified. This lets more IP addresses be configured on a NonStop server instead of configuring multiple LAN adapters. Also, as noted above, the requirement to use the same port number (and a different IP address) in every instance of a cluster is a requirement for those applications that need entity bean and EJB handle failover capabilities. For applications that do not use it, the cluster can be configured with the same IP address for all the server instances, but different port numbers Memory Considerations: Moving QIO to KSEG2 WebLogic Server applications are typically configured with large Java heap sizes, in excess of 128 MB or 256MB. In addition, the JVM and native components (for example, JDBC/MX and the SQL/MX CLI) allocate memory for their own use. Therefore, the overall heap space required in a WebLogic Server instance might be considerably larger than the Java heap space alone. When a process uses the TCP/IP V6 stack transport provider for socket operations (for example, the JVM running the WebLogic Server), it becomes a QIO client. NonStop server QIO shared-memory segments are designed such that when a QIO client process makes a socket request, a portion of the process address space is reserved for the QIO segment. This limits the maximum useable heap size for the JVM running WebLogic Server. 18

19 The size of the QIO segment is determined by the configured physical memory on the system. For a system configured with 4GB of physical memory, 512 MB of QIO segment is used. In addition, 512 MB of the process address space is reserved for QIO segment, if the process is also a QIO client. To overcome the problem of losing address space to QIO, the QIO segment can be configured to be moved to the KSEG2 12 region. This means that the maximum size of the QIO segment is restricted to 128MB (instead of the default 512 MB). Before making the decision to move QIO to KSEG2, check the current maximum QIO segment size in use. If it is within 128 MB (optionally allowing for future requirements that might increase the QIO segment size), QIO can be moved to KSEG2. To determine the QIO segment size in use, use the following SCF command: $ gtacl -p scf SCF - T9082G02 - (30APR03) (01APR03) - 08/18/ :45:04 System \TUXDV1 (C) 1986 Tandem (C) 2003 Hewlett Packard Development Company, L.P. (Invoking \TUXDV1.$DATA02.VENKATR.SCFCSTM) 1-> status segment $ZM00,detail QIO Detailed Status SEGMENT \TUXDV1.$ZM00 State... DEFINED Segment State... STARTED Segment Type... RESIDENT Segment Size MDs in Use Max MDs Used Last Fail Size... 0 Current Pool Size Initial Pool Size Max Pool Size Min Pool Size Current Pool Alloc Max Pool Alloc Current Pool Frags Max Pool Frags The maximum QIO space used in this case is roughly 16MB, which is well below the 128MB limit. So, this can be moved to KSEG2. In fact, the QIO segments in this system have already been moved to QIO based on the value of Segment Type. The value is RESIDENT if QIO is moved to KSEG2 and FLAT_UA, if it has not been moved to KSEG2. Configuring the itp WebServer and Plug-ins When configuring itp WebServer and the plug-in processes, make sure that the tuning information as provided in the Tuning Guide for itp WebServer is followed. A few key points are provided below. For more information on these recommendations, see the itp WebServer Tuning Guide. 1) Make sure that your itp WebServer is configured to listen only on the TCP/IPV6 stack. 2) Make sure that the receive depth for the HTTPD servers are not over-configured. The recommended depth is 50. 3) The maximum number of HTTP processes can be configured to be more than the number of processors to provide for peak usage scenarios. 4) The plug-in processes are single-threaded. Therefore, there is a need to configure more plug-in processes to handle the HTTP requests into WebLogic server. When configuring QIO in KSEG2, carefully consider all the processes that will use the TCP/IP V6 stack and their requirements. As mentioned above, when a process uses TCP/IP V6 stack, it becomes a QIO client. The itp WebServer and the WebLogic Server Plug-in for itp WebServer are configured as Pathway serverclasses. itp 12 KSEG2 is a special region of the privileged address space. 19

20 WebServer is a multi-threaded server but the Plug-in is a single-threaded server. In either case, the serverclasses are typically configured to have enough headroom to handle peak usage scenarios. Under peak usage, the number of instances (and hence the QIO resource consumption by the resources) can exhaust the QIO segment space and lead to application processing failures and potentially processor halts Restricting the List of Ports that Can Be Filtered 13 Two additional defines can be used to restrict ports allowed for TCP/IP V6 stack filtering. TCP/IP V6 stack filtering lets ports be shared by multiple processes. These processes should be running in separate processors (one per processor) except for multicast sockets, where processes can be on the same or different processors and there is no restriction on the number of processes joining the same multicast group. In WebLogic Server configuration, we need TCP/IP V6 stack filtering enabled by default (by specifying the filter key) as described previously. Unfortunately, this means that all listen ports opened by the WebLogic Server instance are then considered for filtering. This affects the execution of the instances, if two instances are wrongly configured with the same IP:port combination. To avoid this situation, the list of TCP/IP V6 stack filtered ports can be restricted using the following defines, one for stream sockets and one for datagram sockets. Both defines have the same format. The define =PTCPIP^FILTER^TCP^PORTS specifies the ports for which the TCP/IP V6 stack software will provide filtering support. The format is as follows: $ add_define =PTCPIP^FILTER^TCP^PORTS class=map\ file=p<startport>.p<endport> where startport and endport are the starting and ending port numbers (inclusive) for which PTCPIP filtering will be provided. Similarly, the define =PTCPIP^FILTER^UDP^PORTS specifies the range of UDP ports for which PTCPIP filtering will be provided. The recommendation is to specify the define =PTCPIP^FILTER^TCP^PORTS such that it excludes the listen ports used by all the instances in the cluster. Set =PTCPIP^FILTER^UDP^PORTS to a range of only one port the multicast port that is specified in the cluster configuration Ephemeral Ports Configuration Ports are used in the TCP/IP protocol communication mechanisms to name the ends of logical connections that carry long-term conversations. For the purpose of providing services to unknown callers, a service contact port is defined by server processes offering the respective services. For example, a server offering the HTTP service is by default expected to service requests arriving at the port 80 on a TCP/IP stack. IANA (Internet Assigned Numbers Authority) defines the port namespace in each TCP stacks into three regions: Well Known Ports are those from 0 through Well Known Ports are assigned by the IANA and on most systems can only be used by system (or root) processes or by programs executed by privileged users. 13 This is needed only if multiple WebLogic Server instances are configured with the same IP address, but different ports (foregoing the failover capabilities for entity beans and EJB handles). 20

21 Examples of well-known ports are the HTTP port (80), HTTPS port (443), TELNET (23), and SMTP (25). Registered Ports are those from 1024 through Registered Ports are listed by the IANA and on most systems can be used by ordinary user processes or programs executed by ordinary users. Examples of registered ports are the TIP port (3160) and X windows system ( ). Dynamic and/or Private Ports are those from through These are also called the ephemeral ports. These ports are assigned by the system for binding unnamed sockets before they can communicate. They can be reused once the application using the sockets closes the socket. In the TCP/IP V6 stack, by default, the ephemeral port range is defined to include both the Registered and Dynamic port ranges; that is, ports ranging from 1024 to These ephemeral port ranges are divided into 16 chunks and each processor gets assigned a chunk of ephemeral ports that is loosely associated with the processor. Depending on the processor a WebLogic Server starts in, its port number might not be associated with the processor it is starting on. This will result in late detection of socket reuse errors if the port number used by WebLogic Server is also in use by another application in the processor where the port number chunk belongs to. For example, consider a WebLogic Server configured to listen on port By default, the MINEPHEREAL PORT is 1024 and MAXEPHEMERAL PORT is So, each processor has a 4000 port range associated with it. ( ) /16 = Port number falls in the chunk associated with CPU 2. CPU 0 has the chunk CPU 1 has the chunk CPU 2 has the chunk and so on. Assume that the WebLogic Server instance (listening on port 10000) was started on CPU 1. Because the WebLogic Server instance was started on CPU 1, the WebLogic Server will not know if a process in CPU 2 was already using the port until the time it tries to accept connections. Because of the way the TCP/IP V6 stack works, the WebLogic Server instance will report a failure only when a client tries to connect, instead of while trying to bind to the port for listening purposes. From that point onwards, the WebLogic Server instance will not be useful for accepting client requests and will have to be forcibly restarted by the nodemanager (if one is configured) after health checks fail. But even on restart, the same can continue. The correct fix for this is to configure the instances properly. The purpose of this section is to provide an overview of port clashes and how they manifest themselves and how to avoid them. Note that these clashes are primarily for WebLogic Servers configured with the same IP address but different ports. 21

22 3.17 Reducing the Minimum Listen Queue Size and Acknowledgement Delay Interval By default, the listen queue sizes for the TCP/IP V6 stack is set to128 and the acknowledgement delay interval is set to 20 milliseconds. Here is an extract from the SCF command output that shows this. The TCP6SAM process name is $ZZTCP by default. 1-> info mon $ZZTCP.#ZPTM0,detail TCPIPV6 Detailed Info MON \HOST.$ZZTCP.#ZPTM0 *TCP Send Space D *TCP Receive Space D *UDP Send Space D *UDP Receive Space D *Delay Ack Time *Delay Ack... ON *Keep Alive Idle *Keep Alive Retry Cnt... 8 *Keep Alive Interval QIO Limit % *Host ID *Host Name... host.domain Program Filename... \HOST.$SYSTEM.SYSnn.TCP6MON *Debug... OFF *Full Dump... ON *All Nets Are Local... OFF *TCP Compat ON *EXPAND Security... OFF *TCP Path MTU... ON *TCP Time Wait Trace Status... OFF Trace Filename... *RFC1323 Enable... ON *TCP Init Rexmit Timeout 1000 ms *TCP Min Rexmit Timeout ms *TCP Listen Queue Min *Initial TTL *Min-Ephemeral-Port *Max-Ephemeral-Port *Max-Priv-Port *TCPCWNDMULTIPLIER... 2 *TCP SACK... OFF *NONSHAREDOUTDIST... OFF *Family... INET Total LNPs Configured.. 0 *TCP Max Rexmit Time ms *TCP Max Rexmit Count The two highlighted values show the defaults. It is suggested that both these values be set to 5. For details on altering these values, refer to the TCP/IP V6 configuration manuals. 22

23 3.18 Logical Network Partitioning One important difference between the classic TCP/IP stack and the TCP/IP V6 stack is that in the TCP/IP V6 stack there is only one manager process ($ZZTCP, typically) with which all interfaces are associated. In classic TCP/IP, multiple TCP/IP processes can be configured each with its own set of interfaces. One of the fallouts of the TCP/IP V6 scheme is that a process using the TCP/IP V6 stack as the provider can use any of the interfaces and hence there is no application isolation. To address this, a new feature is introduced in the TCP/IP V6 stack shipped with the G06.22 RVU. Now the network can be logically partitioned using multiple socket providers (TCP6SAM processes) for a different subset of interfaces as in the classic TCP/IP stack. Alias addresses also can be created in the logical partitions. This new feature lets WebLogic Server administrators partition the network interfaces such that different cluster members in a WebLogic Server cluster use different network partitions, providing for better manageability. Refer to the TCP/IP V6 Configuration Manual in the NonStop Technical Library, which describes the logical network partitioning in detail. 4.0 Fault Tolerance Considerations The nodemanager launch script (nodemanager.sh), provided with the Weblogic Server Toolkit for 8.1 SP3, has been enhanced to let the administrator specify the processor selection per managed server (along with the load balancing schemes). This is helpful in placing the managed servers in different replication groups in nonoverlapping processor lists so that at least one of the primary and backup instances of an HTTP session, Stateful session bean session states, entity beans states, and EJB handles session state is always available. For example, assume that a cluster is defined with two replication groups, Group1 with servers ms1, ms2, and ms3, and Group2 with servers ms4, ms5, and ms6. When the cluster is started up on a four-cpu system, the nodemanager.sh can be configured such that servers m1, m2, and m3 are started in CPUs 0 or 1, and ms4, ms5, and ms6 are started in CPUs 2 or 3. Whenever WebLogic Server needs to choose a secondary server, it gives precedence to a different replication group. Therefore, the secondary state is always available when a CPU is lost (ms1, ms2, and ms3 will choose a server in 2 or 3 and similarly, ms4, ms5, and ms6 will choose a server in CPUs 0 and 1). So, even if a processor with a WebLogic Server instance fails, there is always a secondary server available in one of the remaining processors. 5.0 Performance Considerations 5.1 Number of WebLogic Server Instances per CPU During early implementation efforts it is desirable that the number of instances per CPU be set to one if a full J2EE application (both Web and EJB components) is deployed on the WebLogic Server instances. JSP/Servlet-handling is very CPU-intensive, so reducing the number of WebLogic Server instances per CPU will improve results. Adding more instances per CPU might be beneficial if deployed applications are database intensive but even so, throughput might suffer with more than two very active instances per CPU. 23

24 While an administrative server can also be configured to host applications, it is recommended that the application server in a domain (which has other managed servers besides the administrative server) be used for administrative requirements only. During steady state, the involvement of the administrative server is less, but in a production application, the administrative server provides the monitoring of the WebLogic Server domain. So, for the purposes of estimating the CPU cost, an administrative server can also be considered to contribute to 50% of the CPU cost of a managed server instance. 5.2 SQL/MX Nowait Considerations Performance testing with SQL/MX no-wait facilities has demonstrated that throughput decreases with more than four connections per application server per Connection Pool. This is dependent on the application; for some applications the inflection point might be much higher (for example, 20 connections per pool). Applications should monitor connections being used and measure the performance at various limits. The recommended initial capacity is four. 5.3 Load Balancing WebLogic Server performs load balancing based on home and remote object stubs (called replica aware stubs). Each request gets load balanced based on the replica s knowledge of possible target WebLogic Servers hosting the EJB and routed to an instance. The default load balancing strategies provided by WebLogic Server are limited in functionality. The default strategies are round robin, weighted, and random-load balancing. It is recommended that WebLogic Server applications (particularly those that have lots of DB access) use parameterized load balancing along with DB partitioning. This will help in better throughput. More information on parameterized load balancing is available from WebLogic Server documents on cluster load balancing. 5.4 Disabling the Application Poller Performance testing has shown that running the WebLogic Server in development mode consumes significant CPU resources because of the application polling done in development mode. It is recommended that WebLogic Server instances in production environments use the production mode settings. 5.5 Java Heap Setting It is recommended that the Java Heap for WebLogic Server instances be specified at 256 MB (after moving QIO to KSEG2). This seems to provide a very good starting point. The verbose:gc option of the WebLogic Server Java command line can be used to study the frequency and length of GC operations. This data can be used to tune the heap usage of WebLogic Server instances based on application requirements. Note that the Java VM allocates the maximum required heap for JVM usage at startup. So, the swap space considerations for the JVM will be the maximum Java Heap space specified plus other JVM memory requirements and memory requirements for native components (the primary memory intensive component being JDBC/MX and the SQL/MX CLI). To identify the swap usage of a process or the swap requirements for a CPU, use the NSKCOM command. For example, from the OSS prompt use the following commands to identify the swap usage of all the processes or a particular process: /h/super : gtacl -p nskcom NSKCOM - T5838G05 BASE (22JUL02) - Nov Copyright 1995 Compaq Computer Corporation 24

25 $SYSTEM.SYSTEM.ZSYSCFG KMS.SWAPFILE = 0 $SWAP01.SWAP1.CPU00 KMS.SWAPFILE = 1 $SWAP01.SWAP1.CPU01 KMS.SWAPFILE = 0 $SWAP01.SWAP2.CPU00 KMS.SWAPFILE = 1 $SWAP01.SWAP2.CPU01 KMS.SWAPFILE = 0 $SWAP01.SWAP3.CPU00 KMS.SWAPFILE = 1 $SWAP01.SWAP3.CPU01 KMS.SWAPFILE = 2 $SWAP23.SWAP1.CPU02 KMS.SWAPFILE = 3 $SWAP23.SWAP1.CPU03 KMS.SWAPFILE = 2 $SWAP23.SWAP2.CPU02 KMS.SWAPFILE = 3 $SWAP23.SWAP2.CPU03 KMS.SWAPFILE = 2 $SWAP23.SWAP3.CPU02 KMS.SWAPFILE = 3 $SWAP23.SWAP3.CPU03 KMS.SWAPFILE = 4 $SWAP45.SWAP1.CPU04 KMS.SWAPFILE = 5 $SWAP45.SWAP1.CPU05 KMS.SWAPFILE = 4 $SWAP45.SWAP2.CPU04 KMS.SWAPFILE = 5 $SWAP45.SWAP2.CPU05 KMS.SWAPFILE = 4 $SWAP45.SWAP3.CPU04 KMS.SWAPFILE = 5 $SWAP45.SWAP3.CPU05 NSK-NSK-status su cpu,pin,detail SYSTEM : \SYSNAME LOGTIME : August 18, :09:04 TYPE : (E=Extensible S=Shared O=Owner) (CPU Page size is Bytes) Process Pri User-ID Program File Name C,pin 150 nnn,mm $DATA01.ZYQ00000.Z0007DAS KMSF-BACKED SEGMENTS: (Process Space Guarantee = 1424KB) SEG-ID TYPE SIZE RESERVATION KBYTE PAGES KBYTE B B 1 16 Heap+Global+SRL+Stack 281MB MB TOTAL MB FILE-BACKED SEGMENTS: None As shown, this process currently uses 281MB of swap-space and has two segments allocated. Incidentally, this JVM was started with a heap size of 128 MB, which shows that apart from the Java heap, there is a requirement of roughly 153 MB for process specific data. Note that this swap usage must be checked at a steady state where all the native components used by a JVM have been fully initialized and executing to get the right value for swap usage of a JVM process. Compared to WLS 7.0 and NSJ 3.1, this is a steep increase in the JVM swap usage. Several factors contribute to this. The JVM proper has grown bigger with more memory requirements and the hotspot compiler also needs additional memory for its internal use. Incidentally, the WLS instance for which the swap usage above was obtained was not configured with SQL/MX connection pools. SQL/MX and associated native components have large heap requirements, which also must be considered as part of the overall configuration. WebLogic Server provides a set of attributes to configure the frequency and sample size to check for low memory situations and force a garbage collection. WebLogic Server can be configured with threshold values for detecting 25

26 low memory situations and force Garbage collection on a server. The following attributes of the server element (configured in the Servers->Configuration->Memory Tab in the WebLogic Server Console are used for controlling the thresholds. The attributes under this tab control how often WebLogic Server samples JVM memory and what is the threshold at which WebLogic Server will force a Garbage collection. The default settings sample the JVM every hour and force a garbage collection if the free memory reaches 5% of the free memory when the WebLogic Server instance started. Moreover, when a garbage collection is in progress, the JVM is blocked from performing any other work. So, a good tradeoff must be achieved between the frequency and garbage collection runs. Note that this is in addition to normal garbage collection runs initiated by the JVM when low memory situations are hit during the course of the operation of the JVM. We recommend the following values for the parameters related to memory configuration: GC Threshold : 5 GC Threshold: 5 Sample size: 10 Sampling frequency: <varies> During the initial sampling of application memory requirements, we recommend that the sampling frequency be set to a low value (for example, 6 minutes). Then, during a one hour steady state run, it can be determined whether the configured memory on the server is enough. The output from the -verbosegc JVM option can be studied along with the output from the WebLogic Server monitoring to arrive at the optimal memory configuration for an application. 5.6 Using the WebLogic Server Console to Monitor the Performance of an Application WebLogic Server provides runtime information for monitoring all the WebLogic Server managed components. These can be viewed graphically using the WebLogic Server console or can be queried from a performance monitoring application using JMX facilities. Several key attributes relating to performance, like throughput, memory usage, transaction rate, etc. can be monitored using these attributes. The Server->Monitoring tab provides various subtopics to monitor these attributes. 5.7 Tuning the Java Heap The Java VM used by WLS 8.1 SP3 on NonStop (NSJ 4) is based on the Sun HotSpot compiler server VM implementation. This is significantly different from the NSJ 3.1 JVM which was based on the classic VM implementation. An overview of the HotSpot JVM is available at Not all of the GC mechanisms are available on the NonStop implementation (primarily because of a lack of applicability to the NonStop environment). Refer to the NonStop Server for Java Reference Pages for unsupported heap flags. Primarily, parallel GC and concurrent garbage collectors are not available on the NonStop server and any facilities that depend on these are also not supported. There are several options with respect to Heap management and tuning and tracing. The most common option is verbose:gc which prints garbage collection details (as in prior JVM releases). However, there are several more heap tuning and tracing options in the HotSpot-based JVM. One of the significant enhancements in the HotSpot compiler VM is the garbage collection mechanism. Tuning the Java heap is an art by itself in the Hotspot environment. Another whitepaper that deals with this is available from Even though this whitepaper is targeted to J2SE platform version 1.4.2, many of the key concepts also apply to the J2SE platform. (For additional reading, there is also a document targeted towards the J2SE HotSpot JVM available at 26

27 BEA describes WebLogic Server performance tuning in the WebLogic Service Performance Tuning page available at HP recommends a default minimum and maximum of 256 MB for the Java heap. The maximum for the Java heap can be 256 MB or greater depending on need. As specified in this document, when determining the Java heap space, take care that the address space requirements of other native components linked to the JVM and used by the application are also considered and heap sized accordingly. Also, using a large Java heap space requires that the QIO segment be moved to KSEG2 or otherwise reduced from the default sizes, if possible. 5.8 JMX-Based Custom Monitoring WebLogic Server has a JMX-based management interface. All the runtime state and configuration of WebLogic Server components and objects can be managed using the JMX interfaces. It is possible to create a custom JMX object for managing application components and objects and register it with WebLogic Server. It is also possible to create notification listeners for various management attributes. For example, a notification listener can be registered such that whenever the number of JDBC connections in use reaches a threshold. For additional details, see the WebLogic JMX Programming information available at Many third party products are available, which present the JMX information in useful formats while providing many value additions. 6.0 Additional Tuning Considerations 6.1 Improving Instance Startup Time When an instance of the WebLogic Server starts, several things happen. The WebLogic Server loads its own Java classes and then proceeds to deploy applications that are configured in the config.xml file. Part of this deployment is CPU-bound; typically, the compilation of Java bytecode into machinecode is CPU-intensive. Another part of the deployment is I/O intensive. Applications are typically deployed as archives; Enterprise Archives (archive.ear) and Web Archives (archive.war) are examples of such archives. An application archive is extracted into a server instance-specific directory. This directory will also contain library archives EJB cache and other files that an instance requires. It is the latter part that can be optimized drastically. The goal therefore is to minimize file creates and copying of files. Every application uses some general purpose archives. An example is tdmext.jar, when the Java Toolkit is used. When these jar files are put into the application library, they end up in the application archive. Upon deployment, these files are copied to two locations in the.wlnotdelete directory. This copy is not necessary, and can even be dangerous, because it is unknown which version of the jar (that of the system or that of the originating workstation) will be used when the application is running. You would typically have common jars defined in the WLS server startup CLASSPATH. The jars need not be included in the EAR when the application is built. Application archives are great to transport applications from one system to another. However, WLS needs to extract the application into a server-instance specific directory called.wlnotdelete under the server s root directory. This process is performed upon deployment of an application and when a server is started. In addition to deploying from an archive, WLS provides deployment from a directory that contains the extracted contents of the archive. When WLS deploys the directory, it does not need to copy all the files from the archive. 27

28 Instead, it only copies existing jars to the.wlnotdelete directory (and this can be limited by the procedure described above). Other files that are created are ejb cache files. It was seen that deployment from an extracted directory was almost 50 % faster. 6.2 JMS Considerations JMS lets you store persistent messages in either a disk-based file (called a JMS file store) or in a JDBC-accessible database (called a JMS JDBC store). During performance testing, the following was noted: Using a JMS File Store resulted in less path length but allowed for less concurrency. Using a JMS JDBC Store introduced more overhead but allowed for more concurrency. Thus, using the JMS JDBC Store resulted in more throughput. Using a JMS JDBC Store caused a considerable increase in TMF transactions. Unfortunately, it is almost impossible to address the performance of JMS on WebLogic Server in a generic fashion. Message size, acknowledge mode, persistence mode, and the type of consumer are just a few of the things that can impact performance. With so many variables, it is not possible to extrapolate the performance of another JMSbased application, based on our performance testing. The only way to understand JMS performance is by testing your own application (or a proof of concept). More information on this is available in the WLI Tuning Guide. 6.3 Configuring TMF and DP2 for Use with WLS The Transaction Management Facility (TMF) provides transaction protection and database consistency in demanding online transaction processing and decision-support environments. To furnish this service, TMF manages database transactions, keeps track of database activity through audit trails, and provides recovery methods. Configuring TMF for optimum performance can be a complex task. The optimum configuration depends on the type of transactions in your WLS application and the rate of audit information that is generated. The TMF Planning and Configuration Guide identifies events that suggest the need to reconfigure your audit trails or change transaction limits. You can monitor system activity using the TM View graphical user interface or TMFCOM STATUS and INFO commands to determine if you need to make changes. As an integral part of the operating system, much of the TMF functionality is provided by system components such as the file system and the DP2 disk process. Performance is improved when the primary DP2 disk processes (those for different disks) are spread across processors. 6.4 Tuning the SQL/MX Database and the OSS File System System managers should use their preferred performance analysis tool to monitor the SQL/MX database. System managers also need to carefully watch any files (such as TLOG files) in the OSS File System as those files have a direct performance impact. 6.5 Tuning TLOG Files TLOG files need to be carefully monitored for disk busy access. TLOG files should be placed on separate physical disk drives in order to spread the disk busy load. 6.6 Tuning the Data Cache Tuning the data cache is very application-specific. The goal is to use system memory as wisely as possible, and to perform as much of the disk I/O in cache (instead of going to the physical disk). Follow the standard cache tuning guidelines found in the NonStop Server manuals. 28

29 2005 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. 11/2005