1 How to Monitor Performance Contents 1.1. Introduction... 1 1.1.1. Purpose of this How To... 1 1.1.2. Target Audience... 1 1.2. Performance - some theory... 1 1.3. Performance - basic rules... 3 1.4. Recognizing some common performance problems... 3 1.5. Monitoring, and optimizing, the performance... 4 1.5.1. Tools and mechanisms... 4 1.5.2. Interpreting the request.log... 5 1.5.3. Caching... 7 1.5.4. Analyzing Search... 9 1.1 Introduction CQ5 encompasses several applications, and interacts with several more. Performance (or the lack of it) is one of the first things that your users notice, so as with any application with a user interface, performance is of key importance. To optimize the performance of your CQ5 WCM installation various attributes of the behavior should be monitored. 1.1.1 Purpose of this How To To give a short introduction on what to monitor, and how, when optimizing the performance of your CQ5 implementation. Note This document deals primarily with the mechanics of monitoring performance. Issues such as setting the target metrics of the performance you want to achieve are covered elsewhere. See the CQ5 WCM Project Manager Guide for more information. 1.1.2 Target Audience Power Users System Administrators Project Managers 1.2 Performance - some theory The problems that cause performance issues are often difficult to track down, even when their effects are easy to see. A basic starting point is a good knowledge of your system when it is operating as normal. If you don't know how your environment "looks" and "behaves" when it is performing properly, it can be difficult to locate the problem when performance deteriorates. This means that you should spend some time investigating your system when it is running smoothly and ensure that collecting performance information is an ongoing task. This will provide you with a basis for comparison should the performance suffer. Page 1 of 9
The following diagram illustrates the path that a request for CQ5 content can take - and therefore the number of different elements which can impact the performance. Figure 1.1. CQ5 request - the web-chain Performance is also a balance between Volume and Capacity: Volume the amount of output that is processed and delivered by the system. Capacity the system s ability to deliver the volume. This can be illustrated in various locations throughout the web-chain. Page 2 of 9
Figure 1.2. Capacity vs. Volume There are several functional areas which are often responsible for impacting the performance: Caching Application (your project) code Search functionality 1.3 Performance - basic rules Certain rules should be kept in mind when optimizing performance: 1. Performance tuning must be part of every project. 2. Do not optimize early in the development cycle. 3. Performance is only as good as the weakest link. 4. Always think about capacity vs. volume. 5. Optimize important things first. 6. Never optimize without realistic goals. Note Bear in mind that the mechanism you use to measure performance will often affect exactly what you are trying to measure. You should always try to account for these discrepancies, and eliminate as much of their effect as possible; in particular browser plug-ins should be de-activated wherever possible. 1.4 Recognizing some common performance problems The following lists common performance issues which occur, together with proposals on how to spot and counteract them. Page 3 of 9
Table 1.1. Recognizing common performance problems Area Symptom(s) To increase capacity... To reduce volume... Client High client CPU usage. Install a client CPU with higher performance. Server Network Low server CPU usage. Some clients fast, some slow. CPU usage low on both servers and clients. Browsing locally on the server is (comparatively) fast. Web-server CPU usage on the webserver is high. Application Server CPU usage is high. Repository Cache Simplify (HTML) layout. Upgrade to a faster browser. Improve client-side cache. Remove any network bottlenecks. Increase network bandwidth. Cluster your web-servers. Use a hardware loadbalancer. Improve/optimize the configuration of the client cache. Reduce the "weight" of your web pages (e.g. less images, optimized HTML). Reduce the hits per page (visit). Cluster your CQ5 instances. Search for, and eliminate, CPU and memory hogs (use code review, timing output, etc). High memory consumption. Improve caching on all levels. Low response times. Optimize templates and components (e.g. structure, logic). 1.5 Monitoring, and optimizing, the performance Performance issues may stem from a number of causes that have nothing to do with your website, including temporary slowdowns in connection speed, CPU load, and many more. It may also impact either all your visitors, or only a subset of them. All this information needs to be obtained, sorted and analyzed before you can optimize the performance. If you experience a performance issue: try to replicate it: with one (or preferably more) standard web-browsers, on a different client that you know has good general performance and/or on the server itself (if possible) check whether anything (related to the system) has changed within an appropriate time-space, and if any of these changes could have impacted the performance collect as much information as possible to compare with your knowledge of the system under normal circumstances 1.5.1 Tools and mechanisms The following gives a short overview of some of the tools available for monitoring performance. Page 4 of 9
Note Some of these will be dependent on your operating system. Table 1.2. Tools and mechanisms for monitoring performance Tool Used to analyze... Usage / More information... request.log Response times and concurrency. Interpreting the request.log. truss/strace Page Loads Unix command. Include the misc.truss log level to INFO. CQ5 status monitor Thread dumps Monitor CQ5 requests; including request-type, long-running requests, pending requests. Observe JVM threads. Identify contentions, locks and long-runners. Dependent on the operating system, e.g. kill -QUIT <pid> on Unix/Linux whereas Ctrl-Break on Windows. System calls Identify timing issues. Calls to System.currentTimeMillis() or com.day.util.timing are used to generate timestamps from your code, or via HTML-comments. Note: These should be implemented so that they can be activated / deactivated as required; when a system is running smoothly the overhead of collecting statistics will not be needed. Apache Bench Search Analysis JMeter JProfiler truss/strace, lsof Identify memory leaks, selectively analyze response time. Execute search queries offline, identify response time of query, test and confirm result set. Load and functional tests. In-depth CPU and memory profiling. In depth kernel call and process analysis (Unix). 1.5.2 Interpreting the request.log For full details: http://httpd.apache.org/docs/2.0/programs/ ab.html; basic usage is: ab -k -n <requests> -c <concurrency> <url> Analyzing Search. http://jakarta.apache.org/jmeter/ http://www.ej-technologies.com/ Unix/Linux commands. This file registers basic information about every request made to CQ5. From this valuable conclusions can be extracted. 1.5.2.1 Monitoring traffic on your website The request log registers each request made, together with the response made: 09:43:41 [66] -> GET /author/y.html HTTP/1.1 09:43:41 [66] <- 200 text/html 797ms By totaling all the GET entries within a specific periods (e.g. over various 24 hour periods) you can make statements about the average traffic on your website. 1.5.2.2 Monitoring response times with the CQ5 request.log Page 5 of 9
A good starting point for performance analysis is the request log. You can find the request log at data\author\logs and data\publish\logs for the author and publish instance respectively. The log looks as follows (the lines are shortened for simplicity): 09:43:41 [66] -> GET /author/y.html HTTP/1.1 09:43:41 [66] <- 200 text/html 797ms 09:43:47 [67] -> GET /author/z.gif HTTP/1.1 09:43:48 [67] <- 200 image/gif 141ms 09:43:49 [68] -> GET /author/x.html HTTP/1.1 09:43:49 [68] <- 404 text/html 0ms This log has one line per request or response: The date at which each request or response was made. The number of the request, in square brackets. This number matches for the request and the response. An arrow indicating whether this is a request (arrow pointing to the right) or a response (arrow to the left). For requests, the line contains: the method (typically, GET, HEAD or POST) the requested page the protocol For responses, the line contains: the status code (200 means success, 404 means page not found ) the MIME type the response time Using small scripts, you can extract the required information from the log file and assemble the statistics you want. From these, you can see which pages or types of pages are slow, and if the overall performance is satisfactory. 1.5.2.3 Monitoring search response times with the CQ5 request.log Search requests are also registered in the log file: 22.05.2008 12:27:40 [338] -> GET /author/playground/en/tools/search.html? query=dilbert&size=5&dispenc=utf-8 HTTP/1.1 22.05.2008 12:27:42 [338] <- 200 text/html 1562ms So, as above, you can use scripts to extract the relevant information and build up statistics. 1.5.2.4 Monitoring the numbers and impacts of concurrent users Again the request.log can be used to monitor concurrency and the system's reaction to it. Page 6 of 9
Tests must be made to determine how many concurrent users the system can handle before a negative impact is seen. Again scripts can be used to extract results from the log file: monitor how many requests are made within a specific time span e.g. one minute test the effects of a specific number of users all making the same requests at (as close as possible) the same time; e.g. 30 users clicking Save at the same time 22.05.2008 12:27:22 [333] -> GET /author/libs/personalize/content/statics.close.gif HTTP/1.1 22.05.2008 12:27:22 [334] -> GET /author/libs/personalize/content/statics.detach.gif HTTP/1.1 22.05.2008 12:27:22 [335] -> GET /author/libs/cfc/content/imgs/ logo.rzmnurccynwctpcxyubnitcoibmmw000.default.gif HTTP/1.1 22.05.2008 12:27:22 [335] <- 304 text/html 0ms 22.05.2008 12:27:22 [334] <- 200 image/gif 31ms 22.05.2008 12:27:22 [333] <- 200 image/gif 31ms 22.05.2008 12:27:22 [336] -> GET /author/libs/cfc/content/imgs/ logo.rzmnurccynwctzrxunqbbqtvuucmbrrbuwxz0000.default.gif HTTP/1.1 22.05.2008 12:27:22 [337] -> GET /author/titlebar_bg.gif HTTP/1.1 22.05.2008 12:27:22 [336] <- 304 text/html 0ms 22.05.2008 12:27:22 [337] <- 304 text/html 0ms 1.5.2.5 Timing Statistics for Page Loading To see timing statistics for page loading you can use Ctrl-Shift-U - with? debugclientlibs=true set in the URL. 1.5.3 Caching The following diagram shows the different cache locations that can be used for the various content types. Figure 1.3. What can be cached where? The following can act as a rough guide for target values: Page 7 of 9
Figure 1.4. Cache vs. Uncached - maximum hits / second Although there are many algorithms to ensure that data is retrieved from the source system when appropriate, circumstances can arise where the data residing in a cache is out of date. Retrieving every page individually is the only guaranteed method of ensuring your content is up-to-date, but it is very costly in terms of response, and can indeed cause knock-on effects. This is particularly relevant when using personalized pages, where at least some content of a page is dependent on the user, and the account they used to login. Figure 1.5. Cache speed vs. Data Integrity Page 8 of 9
1.5.3.1 Optimizing your content for cache performance Make sure you use realistic cache settings for the browser cache. If you have disabled the browser cache for development, this may increase traffic and decrease responsiveness. 1.5.4 Analyzing Search First steps to analyzing the search function can be made with Monitoring search response times with the CQ5 request.log. However, once you have determined the response time, you may need to analyze why the request is taking the time it does, and what can be done to improve the response. Page 9 of 9