Common Memory Issues in WebSphere Application Server

Transcription

1 IBM Software Group Common Memory Issues in WebSphere Application Server Ajay Bhalodia, Thomas Ireton & Giri Paramkusham W ebsphere Level II Support April 15 th, 2010 WebSphere Support Technical Exchange

2 Agenda IBM Software Group Brief OutOfMemoryError description Look at two common heap memory users in WebSphere Application Server (W SAS) and potential solutions An example of working an OutOfMemoryError problem WebSphere Support Technical Exchange 2 of 28

3 What is an OutOfMemoryError: There are two main types of memory problems which will throw an OutOfMemoryError error: java heap exhaustion: If a java object cannot be allocated. native memory issue: If there is no more memory available for native code allocations (c/c++ malloc/new). There are other events which can throw an OutOfMemoryError error, such as too many classloaders etc, but these are not very common. The JVM can also throw an OutOfMemoryError if it detects excessive garbage collection. Today, we are only going to look at java heap issues. WebSphere Support Technical Exchange 3 of 28

4 What is a memory leak: A memory leak is when memory has not been released after it is no longer required. The most common leak is putting objects into a collection, and then not removing them: HashMap Hashtable Vector ArrayList Array LinkedList Collection WebSphere Support Technical Exchange 4 of 28

5 What is a memory footprint issue: A memory footprint is when more memory is required by the application, for valid reasons, but there is not enough room for it. This will also cause an OutOfMemoryError, as the memory runs out. It is just not caused by a leak, it is caused by more resources being required than are available (memory). We will look at a couple of common memory issues that occur in WebSphere Application Server (WSAS), and what we can do to eliminate them, or at least reduce the impact of them. Then we will look into a specific memory issue, and what we need to do to determine the code which is causing the problem. WebSphere Support Technical Exchange 5 of 28

6 Example 1: session data: sample I 534,639, com/ibm/ws/webcontainer /httpsession/memorysessioncontext 534,637,792 3 com/ibm/ws/webcontainer /httpsession/sessionsimplehashmap 534,637, array of java/util/hashmap$entry 1,571,024 4 java/util/hashmap$entry 1,284,536 4 java/util/hashmap$entry 1,149,400 3 java/util/hashmap$entry 1,087,048 3 java/util/hashmap$entry In this case, we have just over 500m of memory being used by session data. There are 1,777 sessions alive the largest being just under 1.5m. WebSphere Support Technical Exchange 6 of 28

7 Example 1: session data: sample II 840,140, com/ibm/ws/webcontainer /httpsession/memorysessioncontext 840,138,682 3 com/ibm/ws/webcontainer /httpsession/sessionsimplehashmap 840,138, array of java/util/hashmap$entry 58,712,036 4 java/util/hashmap$entry 56,676,224 4 java/util/hashmap$entry 51,257,426 3 java/util/hashmap$entry 48,087,048 3 java/util/hashmap$entry In this case, we have just over 800m of memory being used by session data. There are only 24 sessions alive, but they are very large. WebSphere Support Technical Exchange 7 of 28

8 Example 1: session data There are two choices we have with session data issues: reduce the number of active sessions reduce the size of the individual session objects To reduce the number of sessions, we can shorten the session timeout. This will affect the users of the application, as it may cause them to loose their login sessions if they go to do something else. We could cluster the application across app servers, which would reduce the number of sessions per app server and therefore the number of sessions on each heap. To reduce the size of session data, the application will have to be changed. Because of the effort involved in this, unless the session data is quite large, we will usually want to try reducing the active number. WebSphere Support Technical Exchange 8 of 28

9 Example 1: session data In our first sample, we have 1,777 active sessions. The largest is 1.5m. In this case, we would want to reduce the number of sessions, as the individual session data is not very large. Reducing the session timeout would probably be the best action here. In the second sample, we only have 24 active sessions. The session data is quite large in this case, averaging over 30m. This is usually way too big, so in this case we would probably have to go into the application and change what is being stored in the session. We would need to see if we could find any specific objects in the session data in the heapdump, so we could get some clues as to which Web Module this session data is from. WebSphere Support Technical Exchange 9 of 28

10 Example 2: cache data 932,086,888 class com/ibm/ws/cache/servercache 757,342,000 com/ibm/ws/cache/cache 747,353,944 array of 17 com/ibm/ws/cache/cacheentry$lruhead 485,580,216 com/ibm/ws/cache/cacheentry$lruhead 196,739,024 com/ibm/ws/cache/cacheentry$lruhead 63,782,048 com/ibm/ws/cache/cacheentry$lruhead 1,252,152 com/ibm/ws/cache/cacheentry$lruhead 485,580,216 com/ibm/ws/cache/cacheentry$lruhead 484,584,120 com/ibm/ws/cache/cacheentry 483,871,800 com/ibm/ws/cache/cacheentry 482,820,640 com/ibm/ws/cache/cacheentry 1,048,400 com/ibm/ws/cache/servlet /FragmentComposerMemento 709,376 2 com/ibm/ws/cache/servlet/fragmentcomposermemento WebSphere Support Technical Exchange 10 of 28

11 Example 2: cache data To reduce the amount of memory used by cache, we can reduce the maximum number of items that are cached. In the Admin Console: Application servers > servername > Dynamic cache service Or if you have a more complicated cache spec: Resources > Servlet cache instances > InstanceN In WebSphere 7.0, there is a new feature where you can limit the cache size in megabytes, not just the number of items cached. This makes configuring the cache much easier. WebSphere Support Technical Exchange 11 of 28

12 Example 3: working an OutOfMemoryError We have an application, RecipeApp, which stores recipes, and allows for a search based on the title of each recipe. We are seeing OutOfMemoryError s now, and we are not sure of why. The recipe list has been growing over time, so this may be of some impact. A heapdump was generated on the OutOfMemoryError. It should be sent (along with other requested files) to IBM support to look at. WebSphere Support Technical Exchange 12 of 28

13 Example 3: working an OutOfMemoryError Following is the heapdump fragment IBM Support would see: 1,073,286,192 com/abc/food/recipeapp 1,073,286,176 com/abc/food/recipelist 1,073,286,160 java/util/hashmap 1,073,286, ,305 array of java/util/hashmap$entry 13,776 java/util/hashmap$entry 13,776 java/util/hashmap$entry 13,776 java/util/hashmap$entry WebSphere Support Technical Exchange 13 of 28

14 Example 3: working an OutOfMemoryError After heap analysis, IBM Support will describe the memory use with something like the following: com.abc.food.recipelist has a HashMap with 383,305 entries, using almost 1gb of memory. Please have your developers check and see if this is expected use for the HashMap, or if there is a potential leak. WebSphere Support Technical Exchange 14 of 28

15 Example 3: working an OutOfMemoryError Since we know that 383,305 entries is a bit high for the number of recipes in our application, we need to investigate into how the HashMap grew so large. So, lets look at com.abc.food.recipelist.java for the HashMap. WebSphere Support Technical Exchange 15 of 28

16 Example 3: working an OutOfMemoryError package com.abc.food; import java.util.*; import com.abc.food.*; public class RecipeList extends List { } public String search(string pattern) { if (exists(pattern)) { return(getvalue(pattern)); } return(""); } WebSphere Support Technical Exchange 16 of 28

17 Example 3: working an OutOfMemoryError There is no HashMap in RecipeList.java. But it does extend another class, we need to look at it: public class RecipeList extends List WebSphere Support Technical Exchange 17 of 28

18 Example 3: working an OutOfMemoryError package com.abc.food; import java.util.*; import com.abc.food.*; public class List { HashMap list = new HashMap(); public void add(string key, String value) { list.put(key, value); } public void remove(string key) { list.remove(key); WebSphere Support Technical Exchange 18 of 28

19 Example 3: working an OutOfMemoryError List.java does have a HashMap: HashMap list = new HashMap(); So, we will need to look at all methods which access the variable list. These methods will also be available from RecipeList. We can see there is an add method which adds into HashMap list: public void add(string key, String value) { list.put(key, value); } We will want to search for all calls to add() from any RecipeList objects, as this is where the data is being added into the HashMap we saw in the heapdump. If there were other methods which also added to the HashMap, we would want to search for all calls to them as well. WebSphere Support Technical Exchange 19 of 28

20 Example 3: working an OutOfMemoryError package com.abc.food; import java.util.*; import com.abc.food.*; public class RecipeApp { RecipeList rl = new RecipeList(); public void initialize() { long i; String s = new String(getRecipe()); for (i=0; i<500000; i++) { String title = Long.toString(i); String recipe = s + Long.toString(i); } } rl.add(title, recipe); WebSphere Support Technical Exchange 20 of 28

21 Example 3: working an OutOfMemoryError In our example, not much searching is needed. In RecipeApp, we can see there are two calls to add(), and one is in a loop. In most real world cases, the searching may take some time. Custom tracing may need to be added to your application, printing out of call stacks, so the developers can see what code path is calling the methods which add elements. Also, the methods which remove are important, as if elements are not removed when they are finished with, this will also cause a leak. WebSphere Support Technical Exchange 21 of 28

22 Example 3: working an OutOfMemoryError System.out.println is a handy way of quickly adding temporary custom trace. It writes out a line to the WebSphere SystemOut.log file, prefixed with a timestamp. This can make finding the leak easier, if the application does not have any existing trace or logging. This quick tracing can be handy to make sure there are the same number of removes as adds. Maybe a certain request does not do the remove associated with its add, which then causes the list to grow forever. Without a trace to be able to count adds and removes, the leak may take a very long time to find. WebSphere Support Technical Exchange 22 of 28

23 Example 3: working an OutOfMemoryError Finding a memory leak can take a long time. But at least from heapdumps, we can find the object which is using memory, then the developers of that code can use debugging techniques to determine the cause. WebSphere Support Technical Exchange 23 of 28

24 Conclusion: IBM Software Group Today we have looked at: Session data memory footprint issues and how to control its memory use DynaCache and how to limit the amount of memory it uses Worked a simple OutOfMemoryError problem and saw how to identify the code causing the leak WebSphere Support Technical Exchange 24 of 28

25 References: "Webcast replay: HTTP Session Management Basics and PD Part I "Webcast replay: HTTP Session Management Basics and PD Part II "Webcast replay: Configuring and Implementing Dynamic Caching in WebSphere Application Server "Webcast replay: Implementing Cache Replication in WebSphere Application Server V6.1 and V7 "New Dynacache Features in WebSphere 7 - Part 1 WebSphere Support Technical Exchange 25 of 28

26 Additional WebSphere Product Resources Learn about upcoming WebSphere Support Technical Exchange webcasts, and access previously recorded presentations at: Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: Join the Global WebSphere User Group Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically: Sign up to receive weekly technical My Notifications s: WebSphere Support Technical Exchange 26 of 28

27 We Want to Hear From You! Tell us about what you want to learn Suggestions for future topics Improvements and comments about our webcasts We want to hear everything you have to say! Please send your suggestions and comments to: WebSphere Support Technical Exchange 27 of 28

28 Questions and Answers WebSphere Support Technical Exchange 28 of 28