WebSphere Application Server Tuning Fundamentals

IBM Software Group WebSphere Application Server Tuning Fundamentals Matt Goshgarian: goshgari@us.ibm.com Nishant Thakkar: thakkar@us.ibm.com WebSphere Support Technical Exchange

Purpose of meeting Create performance awareness Discuss tuning fundamentals Show impacts of performance Teach customers how to test performance themselves WebSphere Support Technical Exchange 2

Agenda IBM Software Group Background on performance Definitions Out-of-box vs. tuned application performance Performance dependencies Key tuning parameters WebSphere Application Server Application Hardware Performance impacts of monitoring WebSphere Application Server Operating System Network Performance tools PMI Tivoli Performance Viewer Advisors Conclusion Questions Links WebSphere Support Technical Exchange 3

Definitions Tuning: IBM Software Group To modify system parameters in order to optimize performance Throughput (requests / second): A quantitative unit to measure performance. The higher the better. Response time (milliseconds): A quantitative unit to measure performance. The lower the better. Trade: A J2EE benchmark application used to measure performance of an application server. Client workload driver: An application that sends requests to stress the server. WebSphere Support Technical Exchange 4

Testing methodology Ensure repeatability No unexpected processes running on system Run on a controlled network Stress WebSphere Application Server to near 100% CPU utilization Test runs: Perform warm-up Test steady state Execute multiple runs of each test Variance is expected WebSphere Support Technical Exchange 5

Background on meeting Out-of-box vs. Tuned application performance Non-tuned vs. tuned throughput performance Througput (higher is better) 500 450 431 requests per second 400 350 300 250 200 150 100 50 0 260 Out-of-box Trade on WAS 6.0.2 Tuned Trade on WAS 6.0.2 66% faster performance after tuning!! Caution Different environments will yield different results WebSphere Support Technical Exchange 6

Performance dependencies Hardware Environment Network Complementing software Application Application server WebSphere Support Technical Exchange 7

CAUTION IBM Software Group Be sure to apply tuning in a test environment first!! WebSphere Support Technical Exchange 9

JVM Set proper heap size Initial heap size / Max heap size Monitor with verbose GC (Garbage Collection) Ideal GC cycle interval should be: No more than 2 seconds in length Longer than 10 seconds apart Increasing size increases GC length but reduces cycle frequency WebSphere Support Technical Exchange 10

JVM Screen shot How to set from Administrative Console: Servers > Application Servers > (server name) > Java and Process Management > Process Definition > Java Virtual Machine WebSphere Support Technical Exchange 11

ORB Pass by reference Disabled by default (Pass by value) Enabled only if EJB client and server are on same WebSphere Application Server instance ex: web container & EJB container use same JVM process Use caution when using this parameter!!! Can cause unexpected behavior Verify that the application is designed to use this parameter An object s reference is modified by both the caller and callee WebSphere Support Technical Exchange 12

ORB Screen shot How to set from Administrative Console: Servers > Application Servers > (server name) > Container Services > ORB Service WebSphere Support Technical Exchange 13

Web container Thread pool Use 5-10 threads per server CPU as starting point (Min and Max) Use Tivoli Performance Viewer to get the optimal values Keep-Alive Increase the value to fit your application s needs Most customers set to unlimited Sessions Number of in-memory sessions Disable session overflow in production Set session timeout Session replication WebSphere Support Technical Exchange 14

Web container thread pool Screen shot How to set from Administrative Console: Servers > Application Servers > (server name) > Thread Pools > WebContainer WebSphere Support Technical Exchange 15

Web container keep-alive Screen shot How to set from Administrative Console: Servers > Application Servers > (server name) > Web Container Settings > Web container transport chains > (select name associated with respective port) > HTTP Inbound Channel WebSphere Support Technical Exchange 16

Web container sessions Screen shot How to set from Administrative Console: Servers > Application Servers > (server name) > Web Container Settings > Session management WebSphere Support Technical Exchange 17

JDBC JDBC driver Type 2 vs. type 4 Type 4 typically performs faster Use type 4 if your vendor supports it http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.ibm.websphere.base.doc/info/aes/ae/rdat_mi nreq.html Connection pools Start off using the default Set max and min connections using Tivoli Performance Viewer to find the optimal amount WebSphere Support Technical Exchange 18

JDBC driver Screen shot How to set from Administrative Console: Resources > JDBC Providers > (provider name) > Data sources > (Data source name) WebSphere Support Technical Exchange 19

JDBC connection pool Screen shot How to set from Administrative Console: Resources > JDBC Providers > (provider name) > Data sources > (Data source name) > Connection pool properties WebSphere Support Technical Exchange 20

Application development Dynamic cache WebSphere specific service Can intercept calls to cacheable servlets Developing application with dynacache can improve performance over 100% Can be used to create a map that is distributed throughout a cluster http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/wel come_nd.html Avoid large objects Causes fragmentation Keep objects less than 200K in size Smaller objects allow the JVM to fully utilize memory Synchronization issues When system won t scale Detect using thread dumps Keep synchronization blocks small Use different objects as locks when possible Session Keep session sizes small Good rule of thumb is less than 100K WebSphere Support Technical Exchange 21

Hardware IBM Software Group If software performance boost still isn t enough, upgrade hardware Number of tiers 1-tier: Application server and database on one machine 2-tier: Application server on one machine, database on another 3-tier: Same as 2-tier, but with HTTP server on a third machine Scaling Vertical Multiple processors across single machine Horizontal Cluster across multiple machines Hyperthreading If supported, will logically increase the number of physical processors WebSphere Support Technical Exchange 22

CAUTION IBM Software Group Remember to apply tuning in a test environment first!! WebSphere Support Technical Exchange 23

Performance impacts of Monitoring Why would you want to always run monitoring? Catch problems before they cause user problems. Gather important documentation during the first failure. Different Levels WebSphere Application Server Operating System Network Application HTTP Server Database Etc WebSphere Support Technical Exchange 25

WebSphere Application Server Monitoring Items Tracing enabled Trace string set to none This is the default Allows tracing to a file to be enabled without restarting the application server. verbosegc enabled Allows for detection of memory leaks. Allows for detection of Garbage Collection related performance problems. Do not run this in production unless you are debugging a problem WebSphere Support Technical Exchange 26

Graph of Tracing enabled Difference between tuned and tracing enabled: 0.93% 500.0 450.0 431.0 427.0 400.0 requests per second 350.0 300.0 250.0 200.0 150.0 259.7 100.0 50.0 0.0 Not Tuned Tuned Trace enabled WebSphere Support Technical Exchange 27

Graph of verbosegc enabled Difference between tuned and verbose GC enabled: 0.47% 500.0 450.0 431.0 429.0 400.0 requests per second 350.0 300.0 250.0 200.0 150.0 259.7 100.0 50.0 0.0 Not Tuned Tuned verbosegc enabled WebSphere Support Technical Exchange 28

Operating System Monitoring Items Process CPU/Memory statistics Tool: ps, perfmon Used to determine if processes are using the expected cpu and memory. Overall CPU/Memory statistics Tool: vmstat, perfmon Used to detect swapping and verify overall cpu/memory usage. Local Network stats Tool: netstat Used to verify the number of connections to and from the machine. WebSphere Support Technical Exchange 29

OS Monitoring Script: Sample only. Not for production use. #!/usr/bin/sh while true; do date >> ps.out echo ------------------------------------------------------------ >> ps.out ps avwwg >> ps.out echo ------------------------------------------------------------ >> ps.out date >> netstat.out echo ------------------------------------------------------------ >> netstat.out netstat -an >> netstat.out echo ------------------------------------------------------------ >> netstat.out date >> vmstat.out echo ------------------------------------------------------------ >> vmstat.out vmstat 5 12 >> vmstat.out echo ------------------------------------------------------------ >> vmstat.out date >> vmstat.out done WebSphere Support Technical Exchange 30

Graph of OS Monitoring Difference tuned and operating system script: 1.93% 500.0 450.0 400.0 431.0 422.7 requests per second 350.0 300.0 250.0 200.0 150.0 259.7 100.0 50.0 0.0 Not Tuned Tuned OS scipt WebSphere Support Technical Exchange 31

Other Monitoring Items Network Tools: Ping, telnet, or a simple HTTP client Used to verify that the network isn t dropping packets, isn t overloaded, and is able to properly route HTTP requests. Database Tools: Vendor specific, and OS level tools Used to verify that the database isn t overloaded and that it s working properly. HTTP Server Tools: Vendor specific, and OS level tools Used to verify that the HTTP server isn t overloaded and that it s properly routing requests. WebSphere Support Technical Exchange 32

Network Monitoring Script: Sample only. Not for production use. #!/usr/bin/sh while true; do date >> ping.out echo ------------------------------------------------------------ >> ping.out ping $1 >> ping.out echo ------------------------------------------------------------ >> ping.out date >> ping.out sleep 45 done Usage: script ip-address This should be run between all sections of the network Example: Client pings HTTP Server and HTTP Server pings Client. HTTP Server pings App Server and App Server pings HTTP Server App Server ping Database and Database pings App Server WebSphere Support Technical Exchange 33

Graph of Network Monitoring Difference tuned and network script: 0.62% 500.0 450.0 431.0 428.3 400.0 requests per second 350.0 300.0 250.0 200.0 150.0 259.7 100.0 50.0 0.0 Not Tuned Tuned Network Script WebSphere Support Technical Exchange 34

Recommended Items Why we recommend some and not others? Performance penalty verse gains in problem determination. Should be enabled during test, and can be enabled in production verbosegc < 2% performance difference Do not enable in production, unless debugging a problem. tracing (with no trace string) Operating System monitoring Network monitoring Database monitoring < 2% performance difference < 2% performance difference < 2% performance difference no tests run with this enabled HTTP Server monitoring no tests run with this enabled Be careful when enabling any monitoring in production Note: Run tests before enabling these in production. Not all systems will have the same performance results. WebSphere Support Technical Exchange 35

Graph of combined items Difference tuned and monitoring items enabled: 1.31% 500.0 450.0 431.0 425.3 400.0 requests per second 350.0 300.0 250.0 200.0 150.0 100.0 259.7 50.0 0.0 Not Tuned Tuned verbosegc, trace, OS script and network script WebSphere Support Technical Exchange 36

WebSphere Application Server Performance Tools Performance Monitoring Infrastructure (PMI) Infrastructure for gathering performance metrics from the application server. Performance Viewers Applications that allow for viewing of PMI data Examples: Tivoli Performance Viewer - Included with WebSphere Application Server Other Tivoli products 3 rd party products Performance Advisors Applications that give tuning advice based on PMI data Examples: Included with WebSphere Application Server - Tivoli Performance Viewer advisor - Runtime Performance Advisor 3 rd party products WebSphere Support Technical Exchange 38

Enabling PMI IBM Software Group Go to the specified location: App server: Servers > Application servers > server_name > Performance > Performance Monitoring Infrastructure (PMI) Node agent: System Administraton > Node Agents > node_agent_name > Additional Properties > Performance Monitoring Infrastructure (PMI) Under the configuration tab Check Enable Performance Monitoring Infrastructure (PMI) Click OK or Apply Save the changes Restart the server WebSphere Support Technical Exchange 39

WebSphere Support Technical Exchange 40

How to setup monitoring Go to: Servers > Application servers > server_name > Performance > Performance Monitoring Infrastructure (PMI) > Runtime Select the desired level Click Apply WebSphere Support Technical Exchange 41

Performance Viewers To properly tune an environment one needs to find performance bottlenecks Tuning settings at random takes longer and can lead to sub-optimal performance. WebSphere Application Server Performance Metrics Performance Viewer/Advisor Tuning Changes WebSphere Support Technical Exchange 42

TPV How to access Go to: Monitoring and Tuning > Performance Viewer > Current Activity > server_name WebSphere Support Technical Exchange 43

TPV How to setup logging Within TPV go to: server_name > Settings > Log WebSphere Support Technical Exchange 44

Tivoli Performance Viewer Table View WebSphere Support Technical Exchange 46

Tivoli Performance Viewer Graph View IBM Software Group SVG must be installed to view the graph. Firefox 1.5 has SVG support. IE Users can download an SVG plugin from Adobe. WebSphere Support Technical Exchange 47

Tivoli Performance Viewer advisor Gives advice on resources Threads Database Connections Data sources Session utilization Can advise on configuration and runtime performance WebSphere Support Technical Exchange 48

Tivoli Performance Viewer advisor Session advice WebSphere Support Technical Exchange 50

Conclusion IBM Software Group Background on performance Performance can increase significantly after WebSphere is tuned for the application and environment. Key tuning parameters Tuning WebSphere is only the beginning. Further tuning can be done to the application environment. Performance impacts of monitoring Doing a little monitoring does not have a heavy performance hit, and can save time fixing problems. Performance tools Monitoring performance metrics enables you to reach optimal tuning with minimal effort. WebSphere Support Technical Exchange 51

Additional WebSphere Product Resources Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/ Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.html Join the Global WebSphere User Group Community: www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistant Learn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/esr_overview_viewlet_swf.html Sign up to receive weekly technical My support emails: www.ibm.com/software/support/einfo.html WebSphere Support Technical Exchange 52

Backup WebSphere Support Technical Exchange 53

Parameters we used to tune Trade Pass by reference = true Web container thread pool = 50 / 50 Max persistent keep-alive requests = Unlimited (-1) Heap size 1024 / 1024 Java security = disabled PMI service = disabled Uninstall default applications WebSphere Support Technical Exchange 54

Our 2-tier environment Server: IBM p630 4-way @ 1.0 GHz CPU 8 GB memory AIX 5.2 WebSphere Application Server 6.0.2 Base Database: IBM p570 4-way @ 1.6 GHz CPU 8 GB memory AIX 5.3 DB2 V8.2 JCC Type 4 (XA) driver Client: IBM x330 Intel P3 Xeon 2-way @ 1.2 GHz CPU 2 GB memory Linux SLES 9.0 WebSphere Support Technical Exchange 55

Other Useful Links Performance Tuning Guide: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.i bm.websphere.base.doc/info/aes/ae/welc6toptuning.html Tivoli Performance Viewer: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?t opic=/com.ibm.websphere.nd.doc/info/welcome_nd.html Advisors: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?t opic=/com.ibm.websphere.nd.doc/info/ae/ae/tprf_tuningentrypoin t.html Trade: https://www14.software.ibm.com/webapp/iwm/web/prelogin.do? source=trade6 WebSphere Application Server Information Center: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp WebSphere Support Technical Exchange 56

Any Questions? WebSphere Support Technical Exchange 57