B M C S O F T W A R E, I N C. PATROL FOR WEBSPHERE APPLICATION SERVER BASIC BEST PRACTICES Ross Cochran Principal SW Consultant
PAT R O L F O R W E B S P H E R E A P P L I C AT I O N S E R V E R BEST PRACTICES INTRO DU CTION This paper will cover the best practices for PATROL for WebSphere Application Server. The paper assumes the reader understands J2EE concepts and terms. New and veteran users of PATROL for WebSphere Application Server will see the basic aspects of J2EE runtime environment management in this paper. First lets cover the basics of a J2EE transaction. An end user will request information from a browser. If the request is for a static HTML document, the request will be handled by the HTTP (Web) server. If the request is for a dynamically made HTML document, the HTTP server will pass the request to the WebSphere Application Server (WAS). The WAS will be running inside of a Java Virtual Machine (JVM). (The JVM will be consuming CPU cycles and memory while handling the request for the dynamically made HTML document). The WAS will pass the request to a servlet. The servlet may ask an Enterprise Java Bean (EJB) for information from a database or the servlet may lookup the information itself. Once all the information is ready for presentation back to the browser-based user, the servlet passes the information to the Java Server Page (JSP). The JSP will pass the dynamically created HTML document back to the HTTP server for eventual delivery to the end user. This is how it works and it works perfectly every time, right? Not exactly, that is why BMC Software developed PATROL for WebSphere Application Server, for when things do not go perfect. This paper will cover how to manage some of the primary components of The WebSphere Application Server with PATROL. It will not cover the other mentioned components of a J2EE transaction such as browsers, networks, and HTTP servers. BMC Software has management solutions for these areas but they are not discussed here. This paper also will not cover advanced J2EE techniques such as profiling, JAVA Management Extensions (JMX), and byte-code instrumentation. Before the individual components of WebSphere are discussed, here are a few limitations of managing WebSphere. These items apply to any J2EE WebSphere Application Server based application. Some J2EE applications are so poorly written, no amount of tuning and management will make them run better You can not improve what you can not measure Make your HTTP (Web) servers do as much work as possible. This will keep unnecessary work off your WebSphere Application Server. In other words, do not have servlets and JSPs serving up static content pages.
JVM memory is like money there is never enough Now we understand the limitations of managing WebSphere. We ask ourselves, what parts of WebSphere can I manage? The remainder of this paper will cover the best practices of using PATROL for WebSphere Application Server to manage The WebSphere Application Server. The primary areas of interest are: Memory How much memory is available or being used by the JVM Servlets These will govern the overall control of a J2EE transaction Threads The JVM allocates a finite number of threads for work EJBs These will control the business logic of a J2EE transaction WAS logs Where to look when a J2EE transaction generates error messages. HTTP Sessions Used to maintain the relationship between a given user s browser and their WAS session.
MEMORY Programming in JAVA is not like many other application programming languages. In the other languages, your code is compiled into machine language; whereas with JAVA, your code is still compiled, but it is not compiled into machine language. JAVA code is compiled into language the JVM will execute. JAVA compiled code does not run natively on the operating system, but on the JVM. This requires the JVM to allocate memory (called the heap ) from the operating system for two reasons: 1. To run the JVM 2. To run JAVA programs Another reason JAVA is different from other programming languages is the J2EE specification allows a JAVA programmer to not manage memory. In other traditional programming languages, memory management (allocating and freeing memory) is the responsibility of the application programmer. However with JAVA, this is the responsibility of the JVM. Keeping the above two reasons for JVM memory management in mind, lets look at how PATROL for WebSphere manages the memory (heap size) of the JVM.
The Total Memory parameter is the total amount of memory allocated by the JVM from the operating system at that moment in time. This parameter is not the maximum possible heap size. The parameter is important because it is all the memory the JVM will have unless the JVM allocates more from the operating system. You also should consider this parm should not exceed the physical memory of the system. If this parm is near the total system memory installed, the operating system itself will not have any memory to run. Access the parameter from the WebSphere icon JVM server icon Server Resources Total Memory. This parameter is updated every five minutes.
The Free Memory parameter is the amount of memory (in MB) of the currently allocated heap size that the JVM has remaining to use. This parameter is important because if the JVM runs out of memory the JVM could stop running. If this parameter is zero, you could have a problem with Garbage Collection (see discussion later in this paper) or have a problem with a JAVA application not freeing up memory/objects (known to many as a memory leak). Important note: The JVM can automatically allocate more memory from the operating system; assuming the JVM has not reached the maximum amount of memory allowable for allocation from the operating system. See the next parm (Percent Memory Used) for more details. For example, the chart below shows the JVM has 115.966 MB of memory free. Access the parameter from the WebSphere icon JVM server icon Server Resources Free Memory. This parameter is updated every five minutes. A traditional graph of this parm will move toward zero followed by a sharp/steep increase. Normally this sharp/steep increase is because Garbage Collection has occurred recently. The result is a graph below that resembles a saw blade.
The Percent Memory Used parameter is the current percentage of memory used inside the JVM based upon the maximum possible heap size. JVMs will allocate memory throughout time, but only allocate up to the maximum heap size. The maximum heap size is specified at JVM startup time and cannot be changed while the JVM is running. So it is important to know of all the memory that is being used based on the possible maximum heap size. Hint 100% equals the maximum possible allocated heap size Not 100% of the currently allocated heap size. For example, the chart below shows 25.150% of the total possible JVM memory is in use. To determine the MB value represented by 25.150%, review your JVM startup parms for the maximum heap size. Then take 25.150% of that maximum heap size to determine the Mbytes currently used by the JVM. Access the parameter from the WebSphere icon JVM server icon Server Resources Percent Memory Used. This parameter is updated every five minutes.
The Garbage Collection Duration is the parameter that displays the average amount of time (over the given discovery cycle) the JVM spent cleaning up unused portions of memory. Remember the JVM is in charge of gathering up and reusing free memory because the J2EE specification frees the JAVA programmer from that task. Garbage collection is the process of the JVM that frees unused portions of memory and returns the portions to the heap size. Garbage collection is good and bad. It is good because it gives the JVM more memory for the JVM heap. It is bad because the JVM tends to slow down J2EE transactions during Garbage Collection. This makes Garbage Collection a balancing act ; if Garbage Collection does not happen, then the JVM runs out of memory. If Garbage Collection happens to often or takes too long, your JAVA programs will slow down. The Garbage Collection Duration should start small and grow over time (this is because initially the JVM does not have a lot of memory to free up). As a rule of thumb, if your JVM is spending more than 5% of the time in garbage collection, you may have a problem. In the chart below, assuming the Garbage Collection Duration is collected every 5 minutes (300 seconds), a good warning level for this parm would be 15 seconds. Access the parameter from the WebSphere icon JVM server icon Server Resources Garbage Collection Duration. This parameter is updated every five minutes.
SERVLETS The Servlet Response Time is the parameter (in milliseconds) that displays average response time of the servlets. This parameter is important because a given servlet could be waiting indefinitely on a backend resource (such as a DB). Some JAVA programmers may not code a finite amount of time to wait before timing out the servlet s request to a DB. Additionally if there are other servlets accessing the same non-responsive DB, then the JVM will be allocating a thread for each of these non-respsponsive servlets. The possible result is the JVM will start to run out of threads. This will eventually result in a frozen web site. (See the Threads portion of this paper for debugging thread problems.) By managing the servlet response time you can prevent a potential thread resource problem in the JVM. Access the parameter from the WebSphere icon JVM server icon Web Modules Select targeted servlet Summary Servlet Response Time. This parameter is updated every five minutes.
The Servlet Total Requests parameter tracks the number of request processed by the servlet during the collection interval. This parameter is important because if you know a given servlet is normally passing requests continually (such as a login servlet) and the number of requests goes to zero, this could be an indication of some problem outside of the JVM (such as the HTTP server out of listeners or a network outage). It could also be an indication a servlet is waiting on a backend DB request. If the chart below was for a normally busy servlet, the time from 11:45 to 13:15 is suspicious (because of no servlet request). It could be all outstanding servlet requests are hanging up waiting on a resource. Check the previous Servlet Response Time parameter from 11:45 to 13:15 for a bottleneck in servlets processing requests. Access the parameter from the WebSphere icon JVM server icon Web Modules Select targeted servlet Select your object Servlet Total Request. This parameter is updated every five minutes.
TH READS The Thread Objects parameter shows the number of objects being used by JVM threads at the end of the collection cycle. This parameter is important because a growing value could indicate an application(s) may not be releasing objects when it is through with them. Eventually this type of problem could result in a JVM memory problem. Remember the JVM allocates memory (heap) for each used object. Access the parameter from the WebSphere icon JVM server icon Server Resources Threads Objects. This parameter is updated every five minutes.
The Thread Percent Maxed parameter shows the average percent of time the usage of the thread pool reached 100%. This parameter is important because the JVM cannot run out of threads. If it runs out of threads, all new servlet and JSP requests will stop. Yes, the JVM can dynamically add more threads, but only to the point of the maximum thread pool size (The maximum thread pool size is set at JVM initiation.) The result is you need to know what portion of the possible thread pool the JVM has been consumed. Increasing the maximum thread pool size is not always the answer. Eventually some number of threads will cause the JVM to request resources (for memory and CPU) that it cannot handle. The JVM will end up requesting resources for threads and not doing any real productive work. This is one of those parameters where a constant value of zero is good. Hint: Before JAVA, using threads was relatively difficult; so programmers stayed away from them. JAVA has made the use of threads much easier; so many JAVA programmers may be new to programming threads in their applications. Watch for a novice JAVA programmer abusing the thread pool. If the value is consistently above 10%, then consider increasing your maximum size parameter of your thread pool. Access the parameter from the WebSphere icon JVM server icon Server Resources Threads Percent Maxed. This parameter is updated every five minutes.
ENTE RPRISE JAVA B EANS You can have a robust web site without Enterprise Java Beans by using your web server, servlets, and JSPs. However, if your web site requires access to a central body of business logic and/or requires access to numerous backend databases, EJBs are for you. EJBs provide a layer of abstraction between your JAVA applications and backend databases. This abstraction layer also provides the coordination of business logic for your JAVA applications. This layer is different from a servlet, in that a servlet provides the coordination for the application program logic of a JAVA application through the WAS. The result is EJBs must perform as well as servlets because of their usage being in the middle of JAVA applications. One of the best parameters to determine overall EJB health is the Bean Method Response Time. The parameter is important because it shows the overall response time (excluding time for bean creation and deletion) of all EJBs. An elongated response time could suggest a poorly coded servlet. This poorly written servlet could be trying to access an EJB (via an entity bean) with multiple calls to satisfy a single DB request. These types of entity bean connections are costly on performance. A better-written servlet could use fewer calls to the EJB (via a session bean) for the same DB information. In the chart below we reached a peak of 500+ milliseconds. Is this good or bad? It depends. Work with your JAVA application programmers to determine if 500 milliseconds are acceptable for their code. Access the parameter from the WebSphere icon JVM server icon Enterprise Beans Beans Summary Bean Method RT. This parameter is updated every two minutes.
The Bean Pool Size parameter shows the average number of objects in the pool over the past collection interval. This parameter is important because the EJB pool size depends on two variables - the number of beans and size of the beans. If your bean pool becomes too large you will deplete your heap size. You might consider placing a cap on your EJB pool size by specifying BEANTYPE=MIN, MAX. Access the parameter from the WebSphere icon JVM server icon Enterprise Beans Beans Summary Bean Pool Size. This parameter is updated every five minutes.
LOGS Servlets have many options for logging. Additionally there are several places logging can occur. Logging can occur in the following WAS supplied logs: Activity System Out Native Standard Out System Error Start Server JAVA application specific Regardless of where logging occurs, logging eventually results in physical I/O to a hard drive. Physical I/O takes exponentially longer to execute than I/O to memory. It is suggested to use logging in development and not in production. If you have to use it in production, use it sparingly (for example never trace a servlet in production).
The Log Growth Size parameter shows the amount of space (in KB) the targeted log has grown over the past discovery cycle. This parameter is important because an unexpected spike in logging could suggest a new JAVA program is being put into production. For example many JAVA programmers perform a common coding technique of inserting a system.out.println() statement to help debug code in development. It is a common mistake to accidentally leave this statement in the code while migrating to production. This accident can cause up to 7X degradation in performance because of the overhead in logging. Another common JAVA programming mistake is a new JAVA application may log meaningless messages. The application programmer may not fully understand the significance of producing meaningful error messages. So be on the lookout for meaningless log messages. Access the parameter from the WebSphere icon Log Monitor Server Pick your log Log Growth Size.
HT TP SE SSIO NS Since browser based sessions are session-less (meaning there is not a permanent session between the WAS and browser), the WAS needs to maintain a record of a browser based user. This record is accomplished via an exchange of a cookie between a stored HTTP session in WAS memory and the users browser. Since the HTTP sessions are stored in memory, this has the potential of affecting the heap size. The trick is to monitor the heap size (covered early in this paper) and monitor the number of live HTTP sessions.
The Live Sessions parameter monitors the current number of live HTTP sessions. This parameter is important because too many HTTP sessions will eventually deplete the WAS of the heap size. The WAS will hold HTTP sessions in memory (hint heap usage) waiting for the HTTP session/user to return. We all know many HTTP sessions (users) will not return to a web site. The result is the WAS may end up holding too many HTTP sessions in memory. To prevent this problem, review your WAS timeout value for inactive HTTP sessions. Access the parameter from the WebSphere icon Server Sessions Live Sessions.
SUMM ARY AND CO NCLUSI ONS Monitor your WebSphere Application Server for a few weeks to determine the normal peaks and valleys of resource utilization. Keep an eye on memory usage and verify Garbage Collection is happening, but not happening too much. Verify your servlets are not taking too long to timeout waiting on files. Threads are a finite resource; verify your JAVA programmers are releasing threads in a timely fashion. EJBs are becoming widely accepted and many companies are still learning how to use them. Remember an EJB can timeout much like a servlet waiting on a backend resource. By monitoring EJB and Servlet response times you can anticipate (and hopefully prevent) a frozen web site. Scrutinize your JAVA logs and only log meaningful messages. In theory a web site should be stateless (this is impractical) requiring minimum HTTP sessions in the WAS, however verify your JAVA programmers try to code as stateless as possible web sites.