Qualogy 2014-08-29 M. Schildmeijer Whitepaper Oracle Exalogic FMW Optimization 1
Inhoudsopgave 1. Preface... 3 2. WebLogic Domain Level... 4 2.1 Domain Enhancements... 4 2.2 JDBC SDP enhancement... 4 2.3 Using AdminServer VIPS for floating IP... 5 2.4 WebLogic Admin channel over InfiniBand private net... 5 2.4.1 HTTP Client channel for AdminServer... 5 2.4.2 T3 Client channels... 6 2.5 Using Embedded Coherence (Several FMW products, SOA suite, IDM)... 7 2.5.1 Configuring Coherence Caches for Dehydrations... 7 2.5.2 Enabling the property... 7 2.5.3 Update SOA JVM settings... 8 2.5.4 Overview of tuning settings... 8 2.6 Cluster replications... 10 2.7. Configure JMS for Exalogic... 12 2.7.1 JDBC Store... 12 2.7.2 JMS and Exalogic... 12 2.8 Other Considerations... 13 2.8.1 Hugepages and Transparent HugePages... 13 3. Oracle Traffic Director enhancements... 14 2
1. Preface For each WebLogic / FMW environment running on Exalogic, some extra optimization work needs to be done in order to get the biggest benefit in terms of resources and performance you can get when combining an Exalogic with an Oracle FMW. This whitepaper describes the different configurations to optimize. This whitepaper focuses on: Java JVM/WebLogic/Exalogic optimizations Domain Level enhancements such as: Cluster replication channels, SDP enhancements JDBC SDP enhancements Domain specific enhancements JMS & TLogs in database (after optimization) SOA Suite specific optimizations Cluster channel session replications Coherence WKA over InfiniBand Separate SOA traffic from Admin traffic HTTP Use of Oracle Traffic Director or use MN s Apache HTTP Proxy SSL termination and offload 3
2. WebLogic Domain Level 2.1 Domain Enhancements On a domain level, the next options have to be set: -Dweblogic.ScatteredReadsEnabled=true Better I/O in high network volume environments ( OSB, Webcenter Portal) -Dweblogic.GatheredWritesEnabled=true -Dweblogic.replication.enableLazyDeserialization=true To be set in setdomainenv # Enable Java Exalogic optimizations EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true -Dweblogic.ScatteredReadsEnabled=true -Dweblogic.GatheredWritesEnabled=true -Dweblogic.replication.enableLazyDeserialization=true" export EXTRA_JAVA_PROPERTIES 2.2 JDBC SDP enhancement Besides configuring JDBC and GridLink for RAC to use InfiniBand, DB connections must use the SDP protocol to improve performance: Complete the following steps: 1. Ensure that you have created the Grid Link Data Sources for the JDBC connectivity on both Exadata nodes 2. The console automatically generates the complete JDBC URL, as shown in the following example: jdbc:oracle:thin:@(description=(address=(protocol=tcp)(host=<>)(port=1521))(c ONNECT_DATA=(SERVICE_NAME=<DB Service>))) 3. In the JDBC URL, replace TCP protocol with SDP protocol. For example: jdbc:oracle:thin:@(description=(address=(protocol=sdp)(host=<>)(port=1521))(c ONNECT_DATA=(SERVICE_NAME=<DB Service>))) 4. Manually add the system property -Djava.net.preferIPv4Stack=true to the startweblogic.sh script. Locate the following line in the startweblogic.sh script: a.. ${DOMAIN_HOME}/bin/setDomainEnv.sh $* b. Add the following property immediately after the above entry: JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true -Doracle.net.SDP=true" By having all these enhanced features, storing JMS artifacts in a JDBC store is more valid than ever. 4
2.3 Using AdminServer VIPS for floating IP The way of setting up an AdminServer VIP for HA and failover will be described in another document. We assume that it is available for use. 2.4 WebLogic Admin channel over InfiniBand private net The private InfiniBand fabric including WebLogic Clusters and Coherence clusters requires a set of IP addresses for all WebLogic Managed Server-to-Administration Server traffic and for cluster communication. These IP addresses are associated with the BOND0 interface. Within Exalogic it is possible to use a private vnet that can be reserved for administrative purposes, such as accessing the AdminServer and its consoles, internal cluster communication and so on. This can be easily achieved by setting all listen addresses to the IP that is bound to InfiniBand We do the same for all managed servers and their listen addresses Trouble is that we can t access the AdminServer url over IPOIB. For this we create separate channels. Also for the managed servers, which have client access, we create channels to connect to the outside world. There are several channels to identify: There are several channels for EoIB interfaces, such as: AdminServer Network Channel HTTP and T3 Network Channels These are needed for internal resources, like AdminServers or to let managed servers communicate with the outside world. You must create two network channels for the Administration Server. The network channels are necessary for routing HTTP traffic and T3 traffic (TCP-based protocol used by WebLogic Server) coming in from an external data center via Ethernet over InfiniBand (EoIB). You must create the following network channels: 2.4.1 HTTP Client channel for AdminServer 1. Log in to the WLS console and click Lock & Edit in the Change Center. 2. In the left pane of the Console, expand Environment, and then Servers. 3. In the Servers tab, click AdminServer (admin). 4. Select Protocols Channels. Click New. 5
5. Enter AdminHTTPClient as the name of the new network channel, select http as the protocol, then click next. 6. Enter the following information in the Network Channel Addressing page: Listen address: < the EOIB IP address >. This address is the floating IP assigned to the Administration Server using the BOND<n> interface. Listen port: 7001 7. Click next, and select Enabled on the Network Channel Properties page and click Finish. 8. Activate these changes. 2.4.2 T3 Client channels Repeat the same steps as above but choose T3 as a protocol. Repeat these steps for all managed servers as well. Example channels configured in a domain AdminServer Managed Servers When started, you see the several interfaces which are listened to: Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.11.26:7001 for protocols iiop, t3, ldap, snmp, http.> <Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "AdminChannel" is now listening on 10.16.160.21:7001 for protocols t3, http.> And you still can use the normal AdminServer address to logon, either hostname or Admin VIP configured. When you configure Oracle Traffic Director, this should be configured using the IPoIB, in every other setup you choose the EoIB, but then you miss all IB enhancements. 6
2.5 Using Embedded Coherence (Several FMW products, SOA suite, IDM) With optimizations in SOA Suite the process state can now reside in Oracle Coherence - which is an in-memory data grid. Coherence uses direct memory access for replicating data. We configure coherence using Unicast; therefore we should disable all multicast entries, located in the setdomainenv. Furthermore, we should set all WKA addresses to use the IPoIB interface, by setting the virtual hostname that is set for the IP address. This is typically the VIP address used for SOA internal traffic (attached to IB interface) -Dtangosol.coherence.wka1=SOAHOST1-PRIV-V1 -Dtangosol.coherence.wka2=SOAHOST2-PRIV-V1 -Dtangosol.coherence.localhost=SOAHOST1-PRIV-V1 SOAHOST1-PRIV-V1 is the virtual host name that maps to the virtual IP where WLS_SOA1 is listening (in SOAHOST1), and so on. 2.5.1 Configuring Coherence Caches for Dehydrations Dehydrations take place in the SOA repository database, but you can use Coherence to offload this kind of work. If you configure the BPEL engine with the CacheEnabled property, the engine runs much fewer reads against database. Because many reads require locks and version checks, eliminating them improves BPEL engine performance. Configuring Oracle Coherence for dehydration requires the following steps: 2.5.2 Enabling the property To enable the cache enabled property in SOA, follow these steps: 1. Log in to Oracle Enterprise Manager Fusion Middleware Control 2. Expand Farm_domain_name, SOA, and then right click soa-infra (server_name) (either one of the servers). 3. Click SOA Administration, and then BPEL Properties. 4. Select More BPEL Configuration properties. 5. Enter CacheEnabled for the property QualityOfService property. 6. Select Apply. Next, set the required server properties for using in-process coherence cache for dehydration. To enable the cache enabled property in SOA: 1. Log into the Administration Console 2. In the Domain Structure window, expand the Environment node. 7
3. Click Servers. 4. Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as hyperlinks) in the Name column of the table. 5. The settings page for the selected server appears. 6. Click Lock & Edit. 7. Click the Server Start tab. 8. In the Arguments field, add the following for all SOA managed servers -Dbpel.cache.localStorage=true -Dbpel.cache.threadCount=20 -Dbpel.cache.cubeInstance.sizeLimit=4g -Dbpel.cache.invokeMessage.sizeLimit=2g -Dbpel.cache.deliveryMessage.sizeLimit=2g -Dbpel.cache.deliverySubscription.sizeLimit=2g 2.5.3 Update SOA JVM settings To optimize behavior with local storage caches and improve performance, update the SOA servers' start parameters to include the following flags: -Djava.net.preferIPv4Stack=true -Xlargepages:exitOnFailure=true -Doracle.xdkjava.exalogic.optimization=true -Xms<n>g -Xmx<n>g ( heapsize used) 2.5.4 Overview of tuning settings Maybe an open door, but every setting you optimize must have a reason, so do not use all these setting just like that. Do it one by one with a group of parameters, and test it. The outcome of your analysis will give you a guideline of what is applicable and what is not. To be set in the setdomainenv, or better use a separate parameter file and reference it to the setdomainenv. Java Optimizations: EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true -Dweblogic.ScatteredReadsEnabled=true -Dweblogic.GatheredWritesEnabled=true -Dweblogic.replication.enableLazyDeserialization=true" export EXTRA_JAVA_PROPERTIES Coherence Optimizations (after Java Optimizations are enabled) # Used with InfiniBand EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa -Dsoa.oracle.home=${SOA_ORACLE_HOME} -Dsoa.instance.home=${DOMAIN_HOME} -Dtangosol.coherence.log=jdk -Dtangosol.coherence.transport.reliable=imb 8
InfiniBand enabling! IMB driver must be enabled for that. Only for Linux x86-64 bit platforms. -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=<location of your trust store>" export EXTRA_JAVA_PROPERTIES BPEL Optimizations. The main parameter is the bpel.cache.localstorage; set it to true and then tune the other parameters to your needs, like cache sizes and worker threads. EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dbpel.cache.localStorage=true -Dbpel.cache.threadCount=20 -Dbpel.cache.cubeInstance.sizeLimit=4g -Dbpel.cache.invokeMessage.sizeLimit=2g -Dbpel.cache.deliveryMessage.sizeLimit=2g -Dbpel.cache.deliverySubscription.sizeLimit=2g" export EXTRA_JAVA_PROPERTIES Note that all the values are examples and can differ from your situation. Always measure, analyze, test and decide! 9
2.6 Cluster replications This applies only for stateful applications such as B2B Console, or SOA/BPM Composer. To enable session replication enhancements for the SOA_Cluster: 1. Shut down the servers in the SOA_Cluster. 2. To set replication ports for a managed server, such as WLS_SOA1: a. Under Domain Structure, click Environment and Servers. Click on the SOA managed server, Configuration tab, Cluster tab. b. In the Replication Ports field, enter a range of ports for configuring multiple replication channels. For example, replication channels for managed servers in SOA_Cluster can listen on ports starting from 8006 to 8009. To specify this range of ports, enter 8006-8009. c. Repeat these steps for all managed servers 3. Create a custom network channel for each managed server in the cluster: Log on to the Admin Console in the Servers table; click the SOA managed server instance. a. Select Protocols, and then Channels, and click New. b. Listen Address: SOAHOST1-PRIV-V1 ( (IPoIB Address ) 4. Listen port: 8006 Remove the default external listen port. 5. Click next, and in the Network Channel Properties page, select Enabled and Outbound Enabled, then click Finish. 6. Under the Network Channels, select ReplicationChannel, the network channel you created for the WLS_SOA1 managed server. 10
7. Expand Advanced, and select Enable SDP Protocol and click Save. You must repeat these steps to create a network channel, each for the remaining managed servers in the SOA_Cluster cluster. Next you should specify the replication channel to you cluster 1. Click the Replication tab. 2. In the Replication Channel field, ensure that ReplicationChannel is set as the name of the channel to be used for replication traffic. 3. In the Advanced section, select the Enable One Way RMI for Replication option, and click save. 4. Activate the changes and restart the managed servers. 5. Add the system property -Djava.net.preferIPv4Stack=true to the start parameters for the WebLogic servers. a. In the Domain Structure window, expand the Environment node and click servers. b. Click the name of the server. c. Enter ReplicationChannel as the name of the new network channel and select t3 as the protocol, then click next. d. Click the Server Start tab. and add the following for the managed servers into the Arguments field: -Djava.net.preferIPv4Stack=true Note that the domain should be restarted. See logs, to see if it succeeded. <Sep 5, 2014 5:53:16 PM CEST> <Notice> <Server> <BEA-002613> <Channel "SOAReplChannel{4}" is now listening on 192.168.11.26:8009 for protocols t3, CLUSTER-BROADCAST, http using SDP for I/O.> 11
2.7. Configure JMS for Exalogic Although not a specific Exalogic enhancement, using all InfiniBand and SDP features storing JMS Artifacts in a database is the preferred way to go. 2.7.1 JDBC Store From here use the common methods to point JMS into JDBC store. However, keep the following in mind: Ensure that the JDBC data source used for the database work exists on the same server instance as the JMS destination the transaction will still be two-phase, but it will be handled with less network overhead. Configure multiple services to share the same store if they will commonly be invoked within the same transaction. Do no use the XA driver for the JDBC Store, because the JMS infrastructure of WebLogic itself is fully equipped for XA JMS often executes direct database operations to invoke its store services within the same transaction, so using a JDBC data source with Logging Last Resource (LLR) might be a good option. It will do two-phase commit in a single local transaction DB operation, this improves the overall transaction performance. 2.7.2 JMS and Exalogic There are some enhancements specifically for Exalogic in doing message compression, either persistency is file based or JDBC bases. The options you have to set are: StoreMessageCompressionEnabled: Enabling Message Store Compression. For Exalogic set this to true to use with the belonging options. MessageCompressionOptions: Along with the option above, choose one of the compression options in it. Note that compression is done by gzip, so you can choose the best compression: GZIP_DEFAULT_COMPRESSION GZIP_BEST_COMPRESSION GZIP_BEST_SPEED LZF PagingMessageCompressionEnabled setting. It's the JMS paging store to perform message body compression on persistent and non-persistent messages. JMS also makes use of the domain wide Exalogic optimizations by using concurrent queues for pending messages. A buffer queue is used for incoming requests which increases throughput by scheduling these requests and prevent locks. 12
2.8 Other Considerations Some other considerations you could think of, either by determining if it s applicable to your situation, it may come out of some load or stress test scenario and analysis. 2.8.1 Hugepages and Transparent HugePages Hugepages allocation can be good for applications which have to allocate large memory blocks. In your Exalogic VM infrastructure design, decide which VM will be used for such applications like WebCenter Portal, and if they need it. In some situation, it can be necessary to reserve a compute node for these environments instead of doing dynamic allocation, and you might consider using Transparent HugePages for these. For more info, see this fantastic article: http://www.qualogy.com/hugepages-for-weblogic/ 13
3. Oracle Traffic Director enhancements In almost every Enterprise Deployment Guide from a Fusion Middleware Product, Oracle uses the Oracle WebTier as a controlling layer for you HTTP calls. In Exalogic, a software loadbalancer is added, called Oracle Traffic Director. Depending on your approach of Cloud infrastructure with one of these components Network topology Segregations of duties and tasks rolled out over several departments, or done by one department overall? Who takes care of what? (Cloud Infrastructure team, Middleware Team, all by one DevOps team) Other solutions than those provided by Oracle (3rd party WebTiers, hardware loadbalancers) Besides the common steps to take in order to use Oracle Traffic Director, there are some specific topics to address. These will be covered in a separate whitepaper. 14