HOL2831 - Beat the Beast: Java Performance Problem Tracking with Open Source Tools

HOL2831 - Beat the Beast: Java Performance Problem Tracking with Open Source Tools Miroslaw Bartecki Solution architect Capgemini Poland CONTACT DETAILS: MIROSLAW.BARTECKI@CAPGEMINI.COM

Table of Contents Preface... 3 What is Zorka?... 4 Hardware and software requirements... 6 Prerequisites... 7 Lab 1. Installing runtime and application... 8 Oracle WebLogic 12c server installation.... 8 Starting Oracle Weblogic 12c Application Server... 10 Installation of Java 6 Enterprise sample application named TRACE... 10 Transaction timeout increase JTA... 17 Lab 2. Installing and configuring Zorka... 19 Zorka agent installation and configuration on Oracle Weblogic Application Server 12c... 19 Zorka collector installation and configuration... 22 Lab 3. Tracing Java problems using Zorka... 24 Tracking first TRACE application problem... 24 Tracking second TRACE application problem... 27 Lab 4. Tracing SQL problems using Zorka... 31 Lab 5. How to reload Zorka server configuration Live... 37 HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 2

Preface This document is a detail description how to monitor and track performance problems related with Java Enterprise applications (JEE) using open source tool. Main tool used to illustrate how performance problems can be tracked is opensource community tool called Zorka. Performance problems tracking is a complex task depended on many factors and can be done through different ways. Because complex nature of problems in JEE applications this lab will only focus on tracking issues related with Java Application Servers components. Main factors at tracked issues will be related with slowness or transaction throughput degradation. Other problems like OutOfMemmory exceptions, heap dumps or garbage collector overload can be track with other standalone tools like Heap Dump Analyzer. Showed at this lab tools can be used to track problems on live production systems used by high volume of users. This was proven on production systems used in commercial projects delivered by Capgemini. Described tools can be configured and adapted to have very small impact on production systems performance. Experience from production systems showed that monitoring system overhead was 1-4 % of overall system performance. This is a big advantage of Zorka tool because it can be used with systems which serve functionalities for hundreds thousands of users. Thumb rule in performance problem tracking is to: define issue in detail analyze affected functionality and components apply appropriate tools and monitor performance factors on safe levels to not overload production system which already has issues In comparison to big vendors who provide complex tools for monitor performance small open source equivalents could fit more adequate, free of charge and with smaller performance overhead. Especially if we want to monitor and track issues in big system with tens of servers license cost from commercial vendors can be high. Free open source tools can be scale out to any number of servers which we will need without impact on licensing costs. Of course free open source has also drawbacks related with tools maintenance, less powerful graphical interface and high skills needed to configure and read results. This lab purpose is to show that with basic Java and operating system knowledge, it is not so hard to track performance issues. To demonstrate common performance problems Oracle Weblogic Apllication Server 12c was used as runtime platform with MariaDB (MySQL) as data store and sample JEE application trace. Sample application was written to demonstrate errors related with performance. Most common development errors met in live systems are related with wrong code or ideas implemented based on requirements. Sometimes libraries or DB functions are used in wrong way. I hope you will enjoy by going through this lab. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 3

What is Zorka? Base tool used to demonstrate how to locate performance issues in this lab is free open source Zorka (http://zorka.io/). To fully understand how this tool is working is good to read below description. It is not needed to read below to complete labs described in this document. Zorka is a flexible java monitoring agent with programmable bytecode instrumentation engine. It is designed to complement existing monitoring systems rather than duplicate their functionality. Zorka agent provides unparalleled flexibility due to its scripting capabilities which makes it unique among comparable products. In addition to core functionalities, set of scripts and zabbix templates distributed with agent provides monitoring capabilities for popular application servers (a.k.a. batteries included ). Zorka agent is working with JDK5, JDK6 and JDK7 and can act in several roles - depending on needs application administrator can enable desired functionalities and configure conventional monitoring systems accordingly. Zorka Stats agent can aggregate performance statistics on arbitrary application components using its instrumentation engine and expose performance counters similar to J2EE statistics via JMX. Tool can log slow SQL-s, EJB-s, HTTP-s to local files, via syslog or to zabbix. Agent is making transaction tracing with code-level visibility. It records application execution (eg. HTTP request) and submits method call tree to zorka collector along with potentially interesting information (eg. HTTP params/headers, SQL queries etc.). Auditing functionality can be easily extended or adjusted using beanshell scripts. Monitored components can be: HTTP - performance statistics, HTTP request tracing, slow request processing logs, error logs SQL - performance statistics, SQL query tracing, slow query logs and error logs (variety of JDBC drivers are now supported, additional can be easily implemented with beanshell scripts) EJB - performance statistics, EJB call tracing, slow query logs, error logs LDAP - performance statistics and tracing for LDAP client requests JMS - performance statistics and tracing for JMS listeners, counters and trace attributes for JMS producers and consumers; Specific components of other application (eg. Jasig CAS, Mule ESB, auditing for various applications etc. can be easily implemented, and some of them are available out of the box. Zorka agent integrates seamlessly with: Zabbix Monitoring System - works in both agent mode and trapper mode, implements low-level discovery; Nagios - simple NRPE integration; syslog - can send information directly to remote to remote syslog servers; SNMP - can send SNMP traps directly to remote management servers; local files - agent can log arbitrary application events it detects in local files (eg. slow SQL queries, EJB errors etc.); ZICO collector - to collect profiling information (application traces); A bunch of popular application servers are supported out of the box: Oracle Weblogic - standard JMX counters, HTTP, EJB tracing and performance counters HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 4

Tomcat 6/7 - standard JMX counters provided by Tomcat, HTTP tracing and performance counters JBoss 4/5/6 - standard JMX counters provided by JBoss, HTTP, EJB, JMS tracing and performance counters JBoss 7, EAP 6 - standard JMX counters provided by JBoss 7, HTTP, EJB, JMS tracing and performance counters, audit Jetty 6/7/8/9 - standard JMX counters, HTTP tracing and performance counters; IBM Websphere 7/8 - standard JMX counters, PMI, HTTP, EJB, JMS tracing and performance counters; Mule ESB - standard JMX counters Note that some common components (eg. SQL, LDAP) are independent of application server and are supported in all above application servers. Administrator must enable appropriate support scripts for those components separately from main application server support script. Beanshell is well known programming language that has been used to make Zorka programmable. Administrator can implement additional beanshell code which will either configure other agent components (eg. methods instrumentation or implement additional queries that can be accessed by monitoring systems. Zorka Spy is an advanced bytecode instrumentation engine using Agent Supplied Aspects technique. It is configured in a manner similar to Aspect Oriented Programming. Application administrator can define arbitrary actions at start and end of method execution, fetch and process arbitrary data (method arguments, current time, current thread etc.) and process data in arbitrary manner. There is a set of predefined components for presenting and processing obtained data: method execution statistics asynchronous signals (traps) intercepting and presenting application objects processing intercepted data and filtering events performing arbitrary actions with bsh functions Zorka Tracer is low-overhead application profiling mechanism suitable to be used in production environments. The following functionalities can be used to control tracer: Changing minimum method execution time and maximum number of methods recorded in a single trace: methods that execute faster than defined time will be omitted (albeit method that throw an exception might still be included) Deciding which traces should be saved and which ones should be dropped: administrator can set minimum trace execution time or programmatically decide if trace should be saved (eg. HTTP requests with HTTP/500 status code should be recorded regardless of how short will they execute Attaching custom attributes to traced methods: any data fetched and processed by Spy can be attached to trace data and later viewed Zorka Trappers functionality allows for asynchronous notifications of external systems. Typically trappers are used in conjunction with Spy, so notifications can be sent when certain methods will be invoked (and certain conditions, defined by administrator will be met). Zorka Intranet Collector (ZICO) collects application performance data from agents all across the company. It collects traces in real time and presents them in user friendly HTML5 application. No special client software and no browser plugins are required. Modern browser is all you need. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 5

Hardware and software requirements This section contains minimal hardware and software requirements to work proceed though this lab. Hardware requirements Processor: Memory: HDD: Screen Resolution: x86/ Intel I3 or greater 8GB 10 GB 1280x1024 Software requirements Operating System with Virtual Box Installed Source materials references http://zorka.io http://www.oracle.com/technetwork/indexes/documentation/ol-1-1861776.html http://www.oracle.com/technetwork/middleware/weblogic/documentation/index.html https://netbeans.org/features/java-on-server/java-ee.html HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 6

Prerequisites Software installed in lab Virtual Machines To demonstrate features related with performance problems tracking below software stack was installed on Virtual Machines: MariaDB database Oracle Weblogic Application server 12c Java 1.7.0.6x 64 bit version At /opt/distributions directory software related with performance measures is present: PERSONDATA.sql script to load sample data to database trace.ear sample JEE application with performance issues inside zico-1.0.7-snapshot.zip server to collect and browse performance data zorka-1.0.7-snapshot.zip agent to work with application server What good to know to perform this lab To fully gain knowledge from this lab is good to have: basic skills with Java Enterprise Application development basic skills with Weblogic application server management basic skills with use Linux operating system Monitoring components architecture Zorka tool high level architecture Zico collector Zico MapDB database 2 Zorka config files Zorka bean shell filters Zico tracing web application Zico server Use zico web application to track issues Zorka monitoring agent 1 Performance issue tracker Monitored Java Application Server Configure filters, scope and time levels for collected data HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 7

Remark - for your convenience and to go straight to performance issues tracking start lab with step: Starting Oracle Weblogic 12c Application Server - page 10 Lab 1. Installing runtime and application To start this lab you need to have Virtual Box Virtual Machine started as described in prerequisites section. Also you have to be logged in to OS using credentials: Username: root Password: oracle In this part you will install and configure: Oracle WebLogic Application Server Oracle MySQL datasource Installation of Java 6 Enterprise sample application named TRACE To login to operating system as root you have to start VM and choose not listed option: And fill credentials like above Username: root Password: oracle Oracle WebLogic 12c server installation. For purpose of this lab will be used Oracle WebLogic Server 12c (12.1.3) Zip Distribution and Installers for Developers. This can be downloaded from http://www.oracle.com/technetwork/middleware/weblogic/downloads/index.html If you are not logged in to operating system please login using above credentials. Open terminal window on linux operating system by clicking in Applications->Terminal window icon on desktop. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 8

Unzip wls1213_dev.zip file located at /opt/distributions directory to /usr/oracle directory using below command: unzip /opt/distributions/wls1213_dev.zip d /usr/oracle In terminal window go to Oracle Weblogic Server directory cd /usr/oracle/wls12130 Export two environment variables $ export JAVA_HOME=/usr/java/jdk.1.7.0_60 $ export MW_HOME=/usr/oracle/wls12130 Install Oracle Weblogic Application Server domain by running $../configure.sh For question about creating domain you should choose option n. Domain will be created later. Setup WLS environment in the current shell. $. $MW_HOME/wlserver/server/bin/setWLSEnv.sh Create a new WLS domain and start WLS. $ mkdir /usr/oracle/wlsdomain $ cd /usr/oracle/wlsdomain $ $JAVA_HOME/bin/java $JAVA_OPTIONS -Xmx1024m -XX:MaxPermSize=256m weblogic.server To question about create default please press on keyboard y and press Enter Put credentials for server domain admin like below: Username: admin Password: Password1 When you will see message like below go to the next step: <BEA-000365> <Server state changed to RUNNING.> Open web browser and navigate to address: http://127.0.0.1:7001/console/ Please wait while console application is deployed. To validate that everything is working correctly you should see login screen: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 9

Starting Oracle Weblogic 12c Application Server Open terminal window on linux operating system by clicking in Applications->Terminal window icon on desktop (if it s not opened yet). Please check that application server is not started, by running command $ ps -aux grep weblogic.server If result is not like root 5452 35.1 15.4 2633412 442568 pts/2 Sl+ 16:23 0:32 /usr/java/jdk1.7.0_60/bin/java - Djava.endorsed.dirs=/usr/java/jdk1.7.0_60/jre/lib/endorsed:/usr/oracle/wls12130/oracle_common/mod ules/endorsed - Djava.endorsed.dirs=/usr/java/jdk1.7.0_60/jre/lib/endorsed:/usr/oracle/wls12130/oracle_common/mod ules/endorsed -Xmx1024m -XX:MaxPermSize=256m weblogic.server You can run application server to test it in foreground by executing in terminal following commands /usr/oracle/wlsdomain/bin/startweblogic.sh This will show server standard output to the this terminal. Do not close this terminal if you close it this will also close application server (java process). Please give 2 minutes to start an application server before you will go with next step. Or alternatively run application server in background nohup /usr/oracle/wlsdomain/bin/startweblogic.sh < /dev/null &> /dev/null & Installation of Java 6 Enterprise sample application named TRACE Install sample application for testing trace of performance problems by opening weblogic console: http://127.0.0.1:7001/console/ Put user and password like at domain creation time: User: admin Password: Password1 HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 10

Open Deployments tab: http://127.0.0.1:7001/console/console.portal?_nfpb=true&_pagelabel=appdeploymentscontrolpage Click install on deployments tab and at the Path input field put: From the list check trace.ear radio button and click Next : On the next page click next with default option Install this deployment as an application. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 11

Leave all defaults and click button Finish. Application will be installed. Wait a while for application installation process to finish. Optional: If you created domain and this is a clean installation please press on last form button Save. If application was installed successfully you will receive message like below: At second step you have to create datasource by going to Services -> DataSources on left menu or by below link click: http://127.0.0.1:7001/console/console.portal?_nfpb=true&_pagelabel=globaljdbcdatasourcetablepage HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 12

On Configuration tab please click button New->GenericDataSource On next tab put datasource data: Name: jdbc/traceds JNDI Name: traceds Database Type: MySQL Form should look like below: Click next to go to the next tab On the next tab choose MySQL Type(4) Driver as it is in default option HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 13

On the next tab left default options like below and click Next On Connection Properties tab put: Database Name: test Hostname: localhost Port: 3306 Database User Name: root Password: Password1 Confirm Password: Password1 HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 14

Click next to go to next tab. Scroll down the page and leave all defaults on this tab (like below) and click button Test Configuration If message is like below click Next to go to next screen, in case of error raise question to presenter. On next form check checkbox next to server on Server list like below and click Finish. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 15

Successful creation will give green message on top of admin page: All changes have been activated. No restarts are necessary. You can test installed application by opening in web browser link: http://localhost:7001/trace/ And next http://localhost:7001/trace/faces/persondata/list.xhtml to see List of persons stored in MySQL test database HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 16

Transaction timeout increase JTA This is an optional step it was already done that you can spend more time on performance problem tracking. For one of the sample queries more than 30 seconds will be needed to complete transaction. Because of this we have to increase parameter responsible for transactions timeout. Application server has to be running. Please open web browser and navigate to Weblogic admin console: http://127.0.0.1:7001/console/ Put credentials: Username: admin Password: Password1 Navigate on left menu to Services-> JTA like on below screen: http://127.0.0.1:7001/console/console.portal?_nfpb=true&_pagelabel=domainconfigjtapage&domainconfigjtaportlethandle=co m.bea.console.handles.jmxhandle%28%22com.bea%3aname%3dmydomain%2ctype%3ddomain%22%29 You will see page like below: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 17

Change parameter Timeout Seconds to 300. After change click button Save on bottom of this form. You should see green message like on below screen: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 18

Lab 2. Installing and configuring Zorka This lab will cover task related with installation and configuration performance problem tracking infrastructure based on open source tool Zorka (http://zorka.io/). Performance tracking components are split to couple parts: Zorka agent responsible for collecting and sending data from connected application server and java application Zorka server part: Zorka collector responsible for storing data send by agent Zorka web application to browse collected data, visualize collection data and track performance problems. Underlying MapDB (http://www.mapdb.org/) database for storing data collected by Zorka Zorka agent is capable to filter and choose appropriate data to be sent to the collector. This mechanism will protect us on gathering too much data and overloading production system by too high data collection on monitoring/profiling purposes. Zorka agent installation and configuration on Oracle Weblogic Application Server 12c Basic agent installation require unzip downloaded.zip package and configure couple basic parameters like agent name or Zorka collector target address. For Oracle Weblogic server, HTTP and EJB monitoring support has been implemented by Zorka. All other appserver-independent functionalities are available if proper script will be loaded (SQL, LDAP, JMX, SOAP etc.). To start installing agent and configure it with Oracle Weblogic Application Server 12c we have to unzip content stored at /opt/distributions/ to application server home directory using below command in new terminal window: unzip /opt/distributions/zorka-1.0.7-snapshot.zip -d /usr/oracle/wls12130/ Naming conventions used in below configuration: ${WL_HOME} = /usr/oracle/wls12130 - Oracle Weblogic directory ${WL_HOME}/server/bin/setWLSEnv.sh - environment variables configuration script If you want just review changes described below in zorka.properties, zico.conf and zico.rpoperties files you can copy and override this files from already prepared location /opt/configs/ Edit zorka.properties file located at ${WL_HOME}/zorka/ and add Oracle Weblogic bean shell script support by using below command: vi /usr/oracle/wls12130/zorka-1.0.7-snapshot/zorka.properties Parameter scripts for Weblogic line 34 should be setup like below (just remove existing # on start of the line and change oracle.bsh to mysql.bsh). scripts = jvm.bsh, zabbix.bsh, oracle/wls.bsh, jdbc/mysql.bsh Save and exit from file (vi command to save and exit Esc -> keys : -> w -> q ->Enter) Above scripts are used to instrument java methods invocations including SQL queries. Next we need to edit startweblogic.sh script for configured server instance and add JVM options similar to below: -javaagent:/usr/oracle/wls12130/zorka-1.0.7-snapshot/zorka.jar/=/usr/oracle/wls12130/zorka-1.0.7-snapshot/zorka -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 19

In our case command and parameters to do changes will look like below: vi /usr/oracle/wlsdomain/bin/startweblogic.sh And add to 78 line (all text in one line no line breaks) JAVA_OPTIONS="-javaagent:/usr/oracle/wls12130/zorka-1.0.7-SNAPSHOT/zorka.jar=/usr/oracle/wls12130/zorka-1.0.7- SNAPSHOT -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder" Copy paste version: JAVA_OPTIONS="-javaagent:/usr/oracle/wls12130/zorka-1.0.7-SNAPSHOT/zorka.jar=/usr/oracle/wls12130/zorka-1.0.7-SNAPSHOT -Djavax.management.bui Save and exit from edition mode. Next edit policy file vi /usr/oracle/wls12130/wlserver/server/lib/weblogic.policy And add on the end of this file below rule: grant codebase "file: /usr/oracle/wls12130/zorka/-" { }; permission java.security.allpermission; Save and exit from edit mode. lder.initial=weblogic.management.jmx.mbeanserver.wlsmbeanserverbuilder" At the end we have to setup few configuration properties in zorka.properties file like below: vi /usr/oracle/wls12130/zorka-1.0.7-snapshot/zorka.properties At line 60 we are changing zorka agent name, this will be used by collector application to identify agents. This is only symbolic name of this agent. Usually used convention is software_name.hostname. zorka.hostname = zorka.localhost At line 73 uncomment and change parameters like below # Uncomment this and set proper IP address to send data to ZICO collector tracer.net = yes tracer.net.addr = 127.0.0.1 tracer.net.port = 8640 This parameters are pointing to the collector ip address and port. Collector will receive on this port data collected by agent. At line 78 comment zabbix server address this will be not used for our test lines should look like below # Uncomment this and set proper address to send data to Zabbix #tracer.zabbix = yes #tracer.zabbix.addr = 127.0.0.1 #tracer.zabbix.port =8640 At line 110 remove # sign and change parameter to catch only http request which are above 200 ms by setting time: http.trace.time = 200 Save and exit from edit mode. Next we have to run Oracle Weblogic Application Server if it is not run or stop and start server if was started. You can stop application server by closing terminal window if it was run in console mode. If it was running using nohup mode please use commands: ps -ef grep nohup kill -9 PID Start Weblogic description HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 20

On the end verify that application server is working correctly by opening sample trace application in web browser: http://localhost:7001/trace/ HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 21

Zorka collector installation and configuration To install collector and related components we need to unzip collector package by executing below command in terminal window: unzip /opt/distributions/zico-1.0.7-snapshot.zip -d /usr/zico Next you have to make.sh executable by run below command chmod 755 /usr/zico/zico-1.0.7-snapshot/zico.sh Next we have to point JAVA_HOME by doing below changes vi /usr/zico/zico-1.0.7-snapshot/zico.conf And change line with JAVA_HOME to: JAVA_HOME=/usr/java/jdk1.7.0_60 Save and exit from edit mode. Next you have to change listen port and address which will be used to serve zico web application by editing a file: vi /usr/zico/zico-1.0.7-snapshot/zico.properties Remove comment from below lines and change port to: zico.http.addr = 0.0.0.0 zico.http.port = 8085 Save and exit from edit mode. Zico web application will be used to browse collected data and identify performance problems. Now you can run zico server by open new terminal window and execute command like below: /usr/zico/zico-1.0.7-snapshot/zico.sh start You should receive message like below: Starting collector...ok. Couple useful information about Zico tool structure: /usr/zico/zico-1.0.7-snapshot/data/ contains MapDB data, in this place tool will store all collected data /usr/zico/zico-1.0.7-snapshot/conf/ contains configuration data db We setup that zico web application used to browse collected data is listening on port 8085. To check that test application trace is monitored and agent is active open web browser and run trace application: http://127.0.0.1:7001/trace/ Click on Show All Person. Link like below Next click Link Next on below page HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 22

Next open new tab in browser and put zico application address: http://127.0.0.1:8085/ You should see page like below with already registered zorka agent: Double click on zorka.localhost to see collected data. This should contain a least data like below: In next lab we will learn how to track performance problems with trace application. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 23

Lab 3. Tracing Java problems using Zorka This lab will demonstrate how to trace performance problems related with Java code. To present how to track Java problems using Zorka we will use trace application deployed at first lab. Trace application has couple of bugs incorporated in source code. Target to this lab is to track them all. There are 2 performance problems incorporated in TRACE JEE application. Before you will start investigate problems make sure that: Weblogic application server is started Zico collector is started Statuses of both can be checked by access to web applications using web browser: Trace application: http://127.0.0.1:7001/trace/ Zico collector: http://127.0.0.1:8085/ or by process checking in terminal window using below commands: $ ps -aux grep weblogic $ ps -aux grep zico If they are not started please start them in terminal window by running command: /usr/zico/zico-1.0.7-snapshot/zico.sh start /usr/oracle/wlsdomain/bin/startweblogic.sh Remark: do not close terminal window because this will close application server process. Tracking first TRACE application problem Open in web browser trace application by entering url: http://127.0.0.1:7001/trace/ Click on Show All PersonData Items link, like on below screen: Next click on Next 10 link like on screen below and wait a minute: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 24

We can observe that next page action is performed slowly. Try to click on Previous link, this one is generating results fast. Let s check what is wrong with this in zico web application. Please open new web browser tab and navigate to zico url: http://127.0.0.1:8085/ Double click on zorka.localhost On traces list put value 5 (red field on screen below) and push enter to filter all request which are below 5 seconds execution time HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 25

Now you can see on the list long execution time of List.xml page like on below screen it takes 12s!!! Let s drill down what s wrong with this execution by click on triangle on the left of trace record: You should see screen like below: We can see that page was rendered from JSF lifecycle with proper result (HTTP code 200). From this perspective everything is ok. Let s now focus on java execution stack. Please double click on trace record red area on screen below: Next use down arrow button to navigate to method which execution time was longest: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 26

Click 3 times on this down arrow. You should see expanded java methods call stack like below: What is most interesting for us is pointed on red. We can see that: There was 1 million of calls to method org.javaone.jsf.persondatacontroller.loopandcount() done by org.javaone.jsf.persondatacontroller.next() method This takes 12 seconds from overall 12, s execution time This is 97,7 % of whole http action time Slow paging is because of that method As we found problematic Java method we can just take source code or decompile it to see what s wrong in this java method. For purpose of this lab this method was called in the loop to emphasize how this kind of problems can be tracked. Java source code snippet looks like below: public String next() { } for (int i = 0; i < 1000000; i++) { } loopandcount(); getpagination().nextpage(); recreatemodel(); return "List"; Tracking second TRACE application problem Close all tabs in web browser if they are open. Open new tab in web browser with trace application by entering url: http://127.0.0.1:7001/trace/ Click on Show All PersonData Items link, like on below screen: Click Edit link next to one of the names on the list HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 27

Make any changes in edit form: Click Save link and wait couple of minutes. As you can observe it is taking long time to render result. If you ve get an error that mean you missed step with increase transaction timeout parameter for JTA (Java Transaction API). Parameter change was described at chapter Transaction timeout increase JTA. You can do this now without application server restart. Let s investigate what s wrong with this action using zico web application. Open in web browser new tab with zico application to browse collected performance data: http://127.0.0.1:8085/ On above screen double click on zorka.localhost If you want to have filtered trace list screen you can limit minimum execution time showed on the list by putting value 40 seconds in filter input field like below and push enter: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 28

On the list should be visible only Edit.xml record. We used this to edit Person Data. What is strange on this there is a lot of Java method calls when we edited this 4 simple fields. It s about 10 million calls. It has to be something wrong with this. Double click on this record to navigate to details. Next click about 6 times to down arrow to jump to longest method executed. You should get list like below: As you can see there is execute query quite fast 339 ms. Expand this executequery() by arrow click on the left. You will get screen like below. As you can see this query produce cartesian result using self-join statement. This as result give 10 million java method calls because JPA is generating results. It can be seen on org.javaone.jsf.persondatacontroller.update() method. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 29

After source code inspection you can see that this method is calling getspeciallist() method from Façade: try { getfacade().getspeciallist(); } catch (InterruptedException ex) { At the getspeciallist() method there is native query Query q = em.createnativequery("select U1.* FROM PERSONDATA U1 JOIN PERSONDATA U2 ON '1'='1'"); We tracked next application problem. Congratulations!!! HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 30

Lab 4. Tracing SQL problems using Zorka This lab will demonstrate how to trace performance problems related with SQL and database code. To present how to track SQL problems using Zorka we will use JEE TRACE application deployed at first lab and MariaDB (MySQL) TEST database instance prepared for this lab purpose. TEST database instance has 1 issue with the structure. Target to this lab is to track this issue. Open in web browser trace application by entering url: http://127.0.0.1:7001/trace/ Click on Show All PersonData Items link, like on below screen: On next form click Create New PersonData Input example data like below and click Save link HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 31

Wait about 10 seconds to get success message like below Open in web browser new tab with zico application to browse collected performance data: http://127.0.0.1:8085/ On above screen double click on zorka.localhost If you want to have filtered trace list screen you can limit minimum execution time showed on the list by putting value 5 seconds in filter input field like below: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 32

You should see collected data from your last PersonData creation like below: As you can see this action take 10 seconds to complete. We have to investigate what was happen wrong with this Person Data creation. Click on small triangle on the left to see request details screen will look like below: As we can see nothing suspicious is in here. We have to investigate deeper. It has to be something wrong with java method call. Double click in web browser to main record for example to 10s You will get trace of java method calls like below: You can navigate on this stack by click on + sign or use arrow to go to method which was executed longest. Click about 6 of times to jump to executeupdate method above 5 seconds execution time. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 33

Double click on this line to see execution details It s simple insert it should not take so much time. We have to check what is wrong with DB. We can also check this by going to main trace tab and select record below HTTP request which is SQL query category. Choose main trace tab and SQL line like below You can see the same data which we saw in Java stack but in SQL category. This is used to trace all queries done through JPA in our test application. Open new terminal window and run below command to open mysql command line: $ mysql -u root --password='password1' test In command line execute show tables to see which tables we have MariaDB [test]> show tables; HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 34

You will get result like below: +----------------+ Tables_in_test +----------------+ PERSONDATA SEQUENCE +----------------+ 2 rows in set (0.00 sec) We should test why insert operation is taking so long. In never version of this DB engine we could insert in SQL command line below statement: MariaDB [test]> EXPLAIN INSERT INTO PERSONDATA(ID, AGE, FIRSTNAME, LASTNAME) VALUES ('1032', '35', 'MIKE', 'TESTER'); Unfortunately on this version of MariaDB(MySQL) explain is not working with INSERTS but only with SELECT. In newer version you will get information that TRIGGER PERSONDATA_INSERT is causing this slowness. You can now drop this trigger by executing below command in SQL command line: MariaDB [test]> DROP TRIGGER PERSONDATA_INSERT; Query OK, 0 rows affected (0.00 sec) After that you can test that this solve situation. Please go to trace application: http://127.0.0.1:7001/trace/ Click Show All PersonData Items link. Next click Create New PersonData link and fill the form with data like below: Click save. Operation should be finished much faster than last time. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 35

Open new tab with web browser and navigate to zico url: http://127.0.0.1:8085/ Double click on zorka.localhost in host list and check that last create action is not present on traces list. This is because we setup a limit to catch only HTTP transaction which are above 200 ms. This time create toked less than 200ms. This can be checked on trace list in SQL category which was not limited by us: Now query take only 2 ms to complete. Congratulations!!! Next issue was tracked down. HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 36

Lab 5. How to reload Zorka server configuration Live There are couple possible ways to made configuration change on Zorka agent be active without restarting whole application server: Use JMX Bean executed by jconsole Use zorka.reload[] via zabbix interface. To use JMX method to reload just change configuration in zorka.properties file and invoke MBean by jconsole. This can be triggered by calling reload() method of zorka:type=zorkacontrol,name=zorkacontrol mbean. Please open new terminal window and in command line execute: /usr/java/jdk1.7.0_60/bin/jconsole From Local process list choose weblogic.server And click Connect button. Next choose Insecure button to connect without SSL. You should see screen like below: HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 37

Please navigate to MBeans tab and you will see MBean list: From this list expand zorka -> ZorkaControl-> Operations like on below screen: Please click reload button to reload configuration. You should see message method successfully invoked. Any changes done in zorka.properties configuration file should be now active. Note that reinstrumenting classes drains permgen on Sun JVM. Agent calculates deltas between reloads in order to minimize number of reinstrumented classes but administrator still needs to ensure that changes don t exceed safe threshold (1000 classes consuming around 50MB of permgen is a good rule of thumb). I hope that you enjoyed entire lab HOL2831 - BEAT THE BEAST: JAVA PERFORMANCE PROBLEM TRACKING WITH OPEN SOURCE TOOLS 38