SOA Solutions & Middleware Testing: White Paper

SOA Solutions & Middleware Testing: White Paper Version 1.1 (December 06, 2013)

Table of Contents Introduction... 03 Solutions Testing (Beta Testing)... 03 1. Solutions Testing Methods... 03 1.1 End-to-End Scenario Testing... 04 1.2 Volume Testing... 05 1.3 Performance Testing... 05 1.4 Endurance Testing... 07 1.5 Security Testing... 07 2. Test Planning and Design... 07 2.1 Types of data transferred... 08 2.2 Usage of the system... 08 2.3 Consumer applications and Integration points... 09 2.4 Message exchange patterns (MEPs)... 09 3. Important Testing Tools... 09 3.1 Testing tools... 09 3.1.1 Apache JMeter... 09 3.1.2 SoapUI... 09 3.1.3 curl... 09 3.2 Monitoring tools... 10 3.2.1 JConsole... 10 3.2.2 JProfiler... 10 3.3 Debugging Tools... 10 3.4 Bug Reporting Tools... 10 4. Continuous Integration... 11 4.1 Tools for Continuous Integration... 11 4.1.1 Source code management System... 12 4.1.2 Build Management System and Automated Tests... 12 4.1.3 Continuous Integration Servers... 12 Summary... 13 2

Introduction Testing is a critical step before end solutions go into production. It ensures that the system performs as expected in the live environment without any errors. A well performing system gains the trust of end users, ultimately resulting in business improvements. This white paper discusses how to test solutions built using SOA. This is also known as Beta testing, which usually involves live data. Brief descriptions on test planning and designing is provided along with other important testing tools such as debugging and monitoring tools. Continuous integration testing procedure is also described giving you a complete picture of the areas involved in SOA solutions and middleware testing. Solutions Testing (Beta Testing) Testing on a developed solution in deployment is an important and a difficult task depending on the technologies used, mainly if the solution is service oriented. Alpha testing happens at the early stages before the product is released where the developers and testers perform component level, integration and system level testing. Once the product is released, solutions testing is done on the actual deployment at the customer environment with live data. This takes place however, before the solution goes live. For solutions testing; verifying response times, testing with large volumes, large files etc. are performed. Solution testing methods include performing end-to-end, volume, performance, endurance and security testing. It is important to plan and design the tests to be performed properly, and that the relevant testing tools are used for accuracy and to increase the testing performance standards. 1. Solutions Testing Methods As a part of solution testing, there are several testing methods that can be performed such as end-to-end, volume, performance, endurance, and security testing scenario testing, to make sure the solution is tested in each of the aspects for best performance. Volume Testing Security Testing Solutions Testing Methods Performance Testing End-to-End Testing Endurance Testing Figure 1 3

1.1 End-to-End Scenario Testing Once the deployment is in place, end-to-end testing is performed to determine if the requirements of the contract are met and everything works as expected. This type of testing is similar to user acceptance testing. While doing end-to-end testing, one should clearly have an idea about all the combinations of user stories that the application is provided with. The following examples can be listed down for end-to-end scenario testing:- Online shopping site User tries to login Searches for items Adds them to a shopping cart Pays for the items purchased Receives confirmation email Healthcare system Registering a patient Feed the system with diagnosed illness and symptoms Monitor medicine intake Monitor patient condition Discharge the patient An API store Registering a user Login to the store Subscribing to an API Retrieve specific keys Use it within an application created for a mobile device Trying out the API 4

1.2 Volume Testing Volume testing is generally done to find out how your application behaves with large amounts of data, and how the system behaves when excessive processing is done. This can be done either with a large number of users in the system, with larger databases or with millions of artifacts in the system. When the end solution is connected to a staging database with live data, one can ensure that the solution is working well under the expected load, and real world artifacts, etc. Through volume testing, we can figure out how large volumes of data can affect throughput/latency, and what issues the users might come across in data access. With large volumes of data in the system, one can find out whether there are any inefficiencies in the system such as slowness in message processing, and whether it would affect other functionalities of the solution. Usually, volume testing is performed for a buffer of around 30% - 50% than the actual peak load. The following scenarios can be performed as volume testing on the end solution depending on the type of the application: Verification of user login when thousands/millions of users exists in the system. When solution is connected to a large database with large amount of data verify message processing time and whether latency increases and throughput decreases. In a banking solution, if the database consists of large volumes of data, monitor the time taken to search customer information. If the solution is an API store, and it consists of thousands of APIs, measure the time taken to load APIs when a search is done. 1.3 Performance Testing Performance testing should always be done based on business needs. As the first step, one should identify what type of usage will take place once the solution is in production; Banking solution - how many accounts will be there in the system Healthcare system - the rate of patients being admitted daily, monthly, etc. Online shopping site - the number of transactions that occur during peak times of the year. Results of the performance tests are used to identify the benchmarks for the solution. It is recommended that the system is tuned properly before a performance test is done. Examples include: Increase the number of open connections Increase memory Disable debug logs 5

Performance testing can be done on areas mentioned below (scenarios can vary depending on the type of your solution. You should have a prior idea about the maximum usage it would have to handle): Online shopping site Verify user login functionality with millions of users. Verify message performance by measuring throughput and latency when millions of transactions happen, i.e. simulate the load during the peak season, such as holidays and festive sales. Healthcare system Simulate patient registration, discharging functionality that might happen in case of situations, such as when a tsunami, earthquake, or explosion takes place. Banking system Simulate the performance of transactions that would occur during peak seasons. To measure performance, different matrices are used. Figure 2 The above-mentioned matrices can be read through tools such as JProfiler and JConsole. To execute performance testing, tools like JMeter and SoapUI can be used. For security related load testing, SoapUI is the best option. JMeter and ApacheBench are tools that can be used to measure metrics like throughput and latency. 6

1.4 Endurance Testing Endurance testing is performed with a significant load to determine how the solution performs over a given period of time with respect to the nature of the solution. Tests are run continuously for an extended time duration, and the system is monitored. Through endurance testing, it can be found whether there are any memory leaks, unusual thread usage or any other weird behaviours in the SOA solution. When a solution is running continuously over a long period of time, i.e. over a year or two, it may hit periods where the usage would reach its peak depending on the type of the solution. Festive seasons and natural disasters are some examples of peak times. Following are steps that can be performed for endurance testing: 1. In case of an online shopping site, a long running test can be setup where the concurrency and frequency of usage can be simulated to match the actual usage during peak times. This can be done using tools like JMeter, Apache Bench or soapui. 2. While these tests are being executed, JConsole can be connected to the running servers, and CPU can be monitored for the usage of memory, threads etc. 3. By executing these tests for months, it can be observed and concluded whether the servers continue to run smoothly without any issues 1.5 Security Testing In any solution, even though the functionalities are working as expected, the application will be useless if any security issues occur, and this will lead to losing the trust of the customers on the solution. Therefore, it is really important that security is thoroughly tested, once the solution is deployed and running flawlessly. When it comes to any solution, you should ensure the following are satisfied with regard to security. Confidentiality - Making sure that the information within the system is only available to the intended parties. Integrity - Ensure that the information is not changed in transit by unauthorized people. Authentication - This is the process of making sure that all parties involved in the process are in fact who they say they are. Authentication can be verified by Username, X.509, SAML assertions, Kerberos tickets, or LTPA binary tokens Authorization - Providing people access to resources in the system, based on their identity. Non-repudiation - Ensuring that the messages have been sent and received by the people who are claiming to have sent and received them. 7

Once the system is up and running, the following steps can be performed to ensure the above: Verification of sensitive information disclosed in the URL. This can be verified by paying attention to the browser caches and in case of Windows, under C:\Documents and Settings\user\Local settings\temporary Internet Files. Verifying that Cross-site scripting (XSS) can be done to ensure authentication and session cookies cannot be stolen, and the accounts cannot be taken over by hackers. To verify this, tools such as WebScarab, XSSer and SoapUI can be used. By verifying possible XSS attacks, integrity and confidentiality can be ensured. Verification of the solution allowing to upload viruses and malware. A validation should be added to check whether only files types with extensions.pdf,.txt,.doc etc are allowed to be uploaded and.exe file uploading is restricted. 2. Test Planning and Design Test scenarios should be constructed based on the related user story. Following aspects should be considered in the process of test planning: Types of data transferred Usage of the system Consumer applications and Integration points Message exchange patterns 2.1 Types of data transferred As the first step, it is needed to identify what type of data is transferred through the SOA solution. This means determining what type of messages come into the system as requests, and what type of responses are sent back from the backend services to the system. For example, if the solution deals with JSON messages, then the test plan should include different types of JSON messages, such as simple JSON messages, and multi-rooted JSON messages. 2.2 Usage of the system When a production system is in place, depending on the user story, one can determine the number of requests that comes into the system and how often they come. Once this number is detected, the test environment can be stress tested with similar numbers. Therefore, in test planning, the identification of the aspects. such as the number of requests received per minute, and concurrency of requests, are needed. 8

2.3 Consumer applications and Integration points The SOA solution might be connected to many third-party applications/servers and different consumer applications. Depending on the type of application, the type of messages and the number of messages being transferred, may vary. Following are the steps that can be performed for endurance testing :- 2.4 Message exchange patterns (MEPs) The SOA solution may support different types of Message Exchange Patterns (MEPs) such as In-Only, In-Out, and Out-Only. Depending on the MEP that is used, test scenarios can be varied. RESTful scenarios can be planned with different types of MEPs. 3. Important Testing Tools There are many testing tools that are useful while working with an SOA solution. Some are used to invoke web services, some are useful for security testing, some to debug the solution and others to monitor the SOA solution. 3.1 Testing tools 3.1.1 Apache JMeter JMeter is an open source tool that can be used for analyzing and measuring performance, and performing REST/SOAP invocations. This tool can be used to automate most user stories, and has the ability for parameterization, allowing one to run the same test from different users. One of the drawbacks of this tools is that it cannot be used to automate security related scenarios. 3.1.2 SoapUI SoapUI is also an open source tool that can be used for web services testing. In addition to functional testing, SoapUI can also be used for performance testing. One of the advantages of using SoapUI is that you can perform security testing using this tool. 3.1.3 curl This is a command line tool used to send or retrieve information with the use of URL syntax. This too is a free tool. This tool supports many internet protocols such as HTTP, HTTPS, FTP, IMAP and many more. curl is not heavily used for load testing and is mainly used for functional testing. 9

3.2 Monitoring tools When the SOA solution is in use, it is important to monitor if it is properly utilizing resources of the server its running on; if the CPU usage is increased when the application is serving requests, and if the program is not overusing threads or memory when it s running in the long run. Some examples of such tools are JConsole and Jprofiler. 3.2.1 JConsole This is a graphical monitoring tool to monitor Java Virtual Machine (JVM) and java applications running on a local or on a remote machine. When the Java based solution and JConsole starts, it will list the application in JConsole. Once connected to the selected java process, CPU usage, memory usage, thread usage, etc., of the application can be monitored. Long running tests can be executed and its behaviour can be monitored using this tool. 3.2.2 JProfiler This tools is a powerful Java profiler that allows to find performance bottlenecks, memory leaks as well as threading issues. With JProfiler, applications that run on the local machine or on remote machines can be profiled. While testing the SOA solution, if you suddenly notice that the CPU usage is increasing and is not responding, you can plug a JProfiler and monitor where it has gone wrong. Which process has caused the issue, and memory leaks if any can be revealed. From the various options available through JProfiler, the exact method that is causing a memory leak, thread leak, and so on, can be determined. 3.3 Debugging Tools There are many ways to debug an SOA solution. One way is to enable DEBUG logs. By enabling debug logs at certain levels, what happens while the message flows through the system can be monitored. Therefore, in such situations, the message body, its properties, and many other information can be monitored. This makes it easier to identify the cause of the issue. If HTTPS traffic needs to be monitored, wire level logs can be enabled for WSO2 products. If the SOA solution deals with SOAP messages, TCPMon can be used to monitor the HTTP traffic that passes through the SOA solution to the backend. Through TCPMon, the message can be captured and the message body, HTTP headers, etc. can be viewed. In addition to these tools, Wireshark can also be used to capture the messages passing through the SOA solution. 3.4 Bug Reporting Tools Once a bug is identified in the solution, there should be a way to record them. When a solution is given for a bug that is reported, a record can be kept for future references as to the actual cause for the bug and the solution. There are some open source bug reporting tools out there that can be used. Some examples are Bugzilla and JIRA. 10

4. Continuous Integration Continuous Integration is a software development practice, which requires developers to integrate code into a shared repository frequently. Each code check-in is verified by an automated build, which allows teams to detect errors quickly and easily. This approach reduces integration problems and allows development teams to work more collaboratively, and build software rapidly. Continuous Integration provides self-sustaining builds, which can be used to get most recent executables. Maintaining a single version control repository enables everyone to check-in the code, find out any clashes between commits, and enable to detect any integration failures early. Dashboards provided by Continuous Integration servers help to keep everyone aware of what is happening. Adhering to Continuous Integration builds leads to a stable solution that works properly and contains few bugs. Figure 3 4.1 Tools for Continuous Integration Any Continuous Integration system requires three types of tools. Source code management system. Build management system and automated tests. Continuous Integration server. 11

4.1.1 Source code management System There are many tools that can be used for source code management. Subversion is the most widely used open source version control system available now. Once the source management system is available, make sure it is available for everyone to check-in or check-out source code. 4.1.2 Build Management System and Automated Tests Maven or Ant can be used as a build management tool. Both tools are open source and available under the Apache licence. To make the build self testing, a suite of automated tests are required to check large parts of the code base. Any Unit framework can be used to write integration or unit tests for SOA solutions. Unit testing frameworks are the starting point for self testing builds. There are quite a number of open source tools available for end-to-testing, such as TestNG, Junit, Selenium, Watir, and HTTPUnit. Maven Surefire plugin or Ant test executor tasks can be used for test triggering. For example, when automating WSO2 products, backend admin services can be used to simulate the artifact deployment, service invocation, and functionalities that can be done through the management console. UI test frameworks can also be used to automate UI oriented admin console functionalities. Selenium has proven to be compatible with WSO2 product user interfaces. If an SOA solution is automated using Jmeter or SoapUI script then integration with maven or Ant can be done by using relevant maven plugins and Ant task libraries. Continuous Integration servers will automatically execute integrated scripts in each build cycle. 4.1.3 Continuous Integration Servers Continuous Integration servers facilitate automation of the software build process. Some of the open source Continuous Integration servers are Jenkins, Cruisecontrol, Apache Continuum and Janky. Continuous Integration servers build executables, and runs automated tests periodically or even after every commit, and reports the result. It s not only for running automated tests but also allows organizations to implement continuous processes of applying quality control. 12

Summary Testing solutions built on SOA is an important and a difficult task. Therefore, it is important how you approach planning and executing the tests for these solutions. As part of solution testing, there are several testing methods that can be performed such as end-to-end, volume, performance, endurance and security testing. These make sure the solution is tested in each of the aspects for best performance. For each of these tests, different testing tools are used, and ApacheBench, JProfiler, JMeter, SoapUI can be named as the most popular. Test scenarios should be constructed based on the related user story. Types of data transferred, usage of the system, consumer applications & integration points, and message exchange patterns should be considered relating to each user stories when planning the tests. Continuous Integration is also an important aspect in SOA solution testing, where it allows to integrate code into a shared repository frequently. Any Continuous Integration system requires three types of tools that can be named as a source code management system, build management system, and automated tests continuous integration server. This approach reduces integration problems and allows development teams to work more collaboratively, and build software rapidly. About WSO2 WSO2 is the lean enterprise middleware company. It delivers the only complete open source enterprise SOA middleware stack purpose-built as an integrated platform to support today s heterogeneous enterprise environments internally and in the cloud. WSO2 s service and support team is led by technical experts who have proven success in deploying enterprise SOAs and contribute to the technology standards that enable them. Check out more WSO2 Whitepapers and WSO2 Case Studies. For more information about WSO2 products and services, please visit or email bizdev@wso2.com