EQUELLA Whitepaper Performance Testing Carl Hoffmann Senior Technical Consultant
Contents 1 EQUELLA Performance Testing 3 1.1 Introduction 3 1.2 Overview of performance testing 3 2 Why do performance testing? 4 3 Common terms in performance testing 4 4 Designing effective performance tests 5 5 Seeding EQUELLA for performance testing 5 6 Building and maintaining performance test scripts 6 7 System performance indicators 6 8 Baselines and benchmarking 7 9 Conclusion 7
1 EQUELLA Performance Testing This white paper provides insights into EQUELLA best practice. It outlines practical ways you can scale, implement, and finetune the EQUELLA Digital Repository in your organisation. 1.1 Introduction Performance testing EQUELLA for capacity planning provides a comprehensive approach for implementing performance testing and using those results to inform decisions about capacity planning. While this is not a comprehensive guide on the wide field of performance testing, you learn how to use and deploy EQUELLA in your organisation, the benefits of performance testing, and the various factors that can affect performance. 1.2 Overview of performance testing With only minor differences and deviations, most of the Information Technology (IT) industry would agree that a standard model for performance testing closely resembles the following: Identify Test Environment Identify Performance Acceptance Criteria Design and Plan Test(s) Configure the Test Environment Implement the Test Design Execute the Test(s) Analyse Results, Report and Retest 1. Identify Test Environment: The first step consists of identifying the physical test environment and the production environment as well as the tools and resources available to the test team. The physical environment includes the hardware, software and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test design and planning and helps you identify testing challenges early in the project. 2. Identify Performance Acceptance Criteria: Identify the response time, throughput, and resource usage goals and constraints. In general, response time is a user concern; throughput is a business concern; and resource usage is a system concern. Additionally, identify project success criteria, which may not be captured by the above criteria; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics. 3. Design and Plan Test(s): Identify key scenarios; determine variability among representative users and how to simulate that variability; define test data; and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analysed. 4. Configure the Test Environment: Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for testing. Ensure that the test environment is set up for resource monitoring as necessary. 5. Implement the Test Design: Develop the performance tests in accordance with the test design. 6. Execute the Test(s): Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment. 7. Analyse Results, Report and Retest: Consolidate and share results data. Analyse the data both individually and as a cross-functional team. Reprioritise the remaining tests and re-execute them as needed. When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration. This model is a good starting point for performance testing, but is too generic. EQUELLA is a complex product, consisting of a web-based Java application, an indexed file store, and a database back-end. Although EQUELLA can be implemented as a stand-alone solution, the reality is that EQUELLA is typically implemented and integrated with a variety of other web-based components. When implemented, EQUELLA shares a database and file systems with many other applications, and is part of a complex IT ecosystem where any individual application s performance may be affected by external factors. When organisations review the EQUELLA environment from this IT ecosystem viewpoint, and understand the power of EQUELLA s scalability via clustering, they may put performance testing into the too hard or too expensive basket. The logic may be that if EQUELLA seems slow, add another cluster node. This kind of ad hoc scaling may work to meet performance requirements, but is inefficient and potentially very costly and wasteful. Performance Testing I 3
2 Why do performance testing? Performance testing is conducted to address one or more risks related to expense, opportunity cost, business continuity or reputation. However, many organisations only conduct testing after a full roll-out is in place. Although this type of approach is better than no testing, possibly the most useful time to test is during the initial pilot phase. The pilot phase of an EQUELLA implementation is a time when organisations are asking questions about how EQUELLA is to be configured and integrated into the IT infrastructure. The common topics of a pilot revolve around initial setup and how to contribute resources into EQUELLA. Although the full end-user community is considered during the pilot, it is usually restricted to a small group participating in the pilot. The full range of end-users and their needs is usually a tangential topic. However, considering the final and full community of users is quite valuable, it can help not only with infrastructure planning, but the questions asked when designing performance tests may provide additional considerations when designing collections. Other benefits to early-stage performance testing include: The benchmarking of components in order to assess performance increases due to changes and upgrades is a worthwhile practice. Therefore benchmarking a single node of EQUELLA will give a good idea of how much extra capacity will be brought online by activating additional nodes. Testing a configuration of multiple nodes on a single physical piece of hardware helps reveal how many nodes may be concurrently run on one machine. The resultant data from testing can be used to help predict the likelihood of user satisfaction or dissatisfaction, based on performance thresholds. Comparing different configurations to determine which works best for the application and the organisation. Comparing the performance characteristics of different versions of EQUELLA. If the overall end-to-end web application is underperforming, being able to quickly performance test one component helps isolate and pinpoint problems in infrastructure. Analysing results of tuning different components, like databases and memory settings. 3 Common terms in performance testing Performance testing is a very broad field, and it is helpful to define different aspects of performance testing. Performance testing may be broken down into four broad categories: n Performance testing this type of testing determines or validates the speed, scalability, and/or stability characteristics of the system or application under test. Performance is concerned with achieving response times (a user concern), throughput (a business concern), and resource usage levels (a system concern) that meet the performance objectives. n Load testing this is a subcategory of performance testing, which is focused on determining or validating performance characteristics of the system or application under test when subjected to workloads and load volumes anticipated during production operations. n Stress testing this is another subcategory of performance testing, which is focused on determining or validating performance characteristics of the system when subjected to conditions beyond those anticipated during production operations. These tests are designed to determine under which conditions an application will fail, how it will fail, and what indicators can be monitored to warn of an impending failure. n Baselines this is the process of running a set of tests to capture performance statistics for the purpose of evaluating the effectiveness of subsequent performance-improving changes to EQUELLA. Baselining may be created for a standalone system, or a multi-component web application. Those conducting a performance test on EQUELLA must keep in mind which type of test is being performed for each phase of an EQUELLA implementation. In the pilot phase, conducting performance testing and baselining will inform decisions regarding appropriate hardware and software configurations for phase 2 deployments. Once a limited-production rollout is complete, a second baseline should be performed against the entire web-application, including any integration. At this stage performance testing as well as load testing may also be carried out to ensure that the system meets the performance expectations of the user community, and has sufficient capacity for short-term growth. Once EQUELLA is deployed into full production, load and stress testing not only verify performance but help define the limits of the implemented system. 4 I Performance Testing
4 Designing effective performance tests When designing performance tests for EQUELLA it is important to profile the range of users by segmenting them according to their requirements and thresholds of satisfaction. Users may be classified as students, teachers, librarians, content administrators or EQUELLA administrators. In EQUELLA pilot implementations, one of the topics discussed is the role of users in terms of security access. This same information is also relevant to performance testing. When building user profiles, consider when and how users will be accessing content. Students will typically access through a course, delivered via a Learning Content Management System s direct link. This requires no loading of EQUELLA s user interface or searching. Although an individual student s demands on EQUELLA will be the lightest; en masse, students will tax the system the most. Typically, students will access content on EQUELLA in bursts with access peaking during certain hours. Students are also highly sensitive to response times and download speeds and are therefore the most difficult audience to satisfy. Content contributors, by virtue of a longer and more interactive experience with EQUELLA, may be less sensitive to download speeds, but are highly sensitive to availability and the response time of the system as they navigate through multipage form contributions. The period in which they access the system may be during off-peak, as courses are typically built and undergo workflow between semesters. Since there are far fewer content contributors than students, load is not an issue, but they will not tolerate outages resulting in the loss of work during their course-building sessions. Administrators typically perform tasks related to searching and running reports against content which can generate high loads. Typically administrators perform their tasks during non-peak hours but will be sensitive to both response time and availability. Each of these roles has a different load profile which may be activated at different times of the day, and some may overlap. Understanding user profiles and their tasks greatly aid in designing performance tests, and also inform how EQUELLA should be implemented and deployed. The following is a list of some important considerations when profiling users: The number of users able to log in over a specified period. Page views per period. A page view is a request that includes all dependent file requests (images, CSS files, for example). User sessions per period. A user session is the sequence of related events (searching, downloading files, contributions) originating from login. The session duration. This represents the average amount of time a user session is expected to last, measured from login to logout, and includes the time the user may pause when navigating from page to page. Interaction speed. This represents user think time, page view time and user delay. This should have some variability because each user will interact with EQUELLA at a different rate. 5 Seeding EQUELLA for performance testing Once user profiles have been identified, the next step in designing effective performance tests that will return consistent results is to configure EQUELLA correctly. Testing against live data and active institutions in EQUELLA will not yield usable results, because there may be great variability in the content being downloaded. For example, collections may contain documents, image files and other multimedia files. Since EQUELLA is a web application, there are limits to the number of allowable concurrent connections. If a performance test measuring latency randomly selects different files to download, it is possible to exceed the number of allowable connections while the current connections are still downloading large files. This type of scenario could falsely record a high latency and slow performance. In order to accurately measure the number of transactions per second for EQUELLA, it is important to seed EQUELLA collections with a number of similar-sized files that approximate the average size of files in live collections. Strategies to accurately seed collections include: Determine the average size of files contributed to EQUELLA and use them to build exemplar collections to test against. Ensure there are a sufficient number of different test user accounts to accurately simulate the number of users expected to connect to EQUELLA during a test. Users must have adequate permissions to perform the actions specified by the test. Configure the exemplar collections with the same metadata schema, contribution wizards, power searches and save scripts. Ensure that the performance requirements and goals represent the needs and desires of the users. Performance Testing I 5
6 Building and maintaining performance test scripts Once user profiles are developed and EQUELLA exemplar collections are properly seeded, there will be enough information to start identifying actual step-by-step activities to load test. A user visit to EQUELLA comprises a series of related requests known as user sessions. Rather than trying to model strictly on the basis of concurrent users, it may be more useful to base the model on user sessions. The only exception to this is with students accessing EQUELLA through a Learning Management System (LMS), where groups of students will load EQUELLA in bursts with similar requests, in which case concurrent sessions is also a valuable metric. Performance test scripts need to take this process into account. If a test is comprised of students logging in and directly accessing a known file attachment and then logging out, then the test should measure not only the number of user sessions over a given period of time, but also the peak number of concurrent users EQUELLA can serve. It is also important to have separate tests for accessing EQUELLA directly and through an LMS. Each component of an end-to-end web application will have an impact on performance. Performance test scripts should also validate the resources retrieved. The impact of not validating the retrieved resources is that none of the performance metrics gathered will have any meaning, because it will be unclear if the resources can be fully accessed. Therefore it is critical to completely retrieve the resources when performance testing. Creating executable performance tests are extremely tool-specific. Regardless of the tool being used, creating a performance test typically involves taking a single instance of the test script and gradually adding more instances and/or scripts over time, thereby increasing the load on EQUELLA. It is important not to allow the choice of test tool to influence the test design. It is better to design tests on the assumption that it can be executed rather than avoiding certain tests based on the assumption that the tool cannot perform the test. While building initial test scripts, it may be helpful to manually perform each activity, recording each step on paper from login to logout. Try to find the natural paths of navigation to build each scenario. Interview a sample of end-users to find out how long they may wait on a page, how they browse and search for content, how they contribute and what their maximum thresholds for response times are. During the early stages of development and testing, user data and variances are most often estimated based on expected usage and observation of users working with similar applications. These estimates can be enhanced and revised when more empirical data becomes available. User feedback should be constantly solicited to improve and update the performance tests. 7 System performance indicators The key to determining the objectives of a performance testing effort is to identify changes, potential risks and opportunities for improvement. The simplest way to determine and record acceptance criteria is to simply ask the different stakeholders what constitutes acceptable performance for them. This helps you capture and quantify resource usage targets and thresholds. Generally speaking it is not acceptable for the performance tester to determine the targets and thresholds, but only to capture data and compare test results to targets and thresholds set by the different stakeholders. Also remember that the stakeholders are not simply the end-users of EQUELLA, but also the system support teams. Even if a performance test succeeds for an end-user, the system support teams may set a threshold of network or processor usage that is different to this threshold. It is generally accepted that performance will degrade significantly after a machine s processor usage exceeds 80 percent. Processor, memory, network and disk I/O usage are important performance acceptance criteria that should not be overlooked. Performance tests may be executed in such a way to find out the number of concurrent users or the number of user sessions before they exceed processor thresholds for a single EQUELLA node. Armed with this information and the target number of users EQUELLA is required to support, organisations can determine the number of EQUELLA nodes for each phase of a rollout. 6 I Performance Testing
8 Baselines and benchmarking As previously stated, a baseline is the process of running a set of tests to capture performance metric data, to evaluate the effectiveness of future performance-improving changes against a known set of metrics. Critical to baselining is the need to record the exact configuration, including all configuration options and architecture components prior to the execution of a baseline test. This provides a standard comparison against future changes. Organisations may have multiple baselines of EQUELLA: as a single-node, non-clustered system; in a multi-node cluster with a load balancer; and accessed directly or through an LMS. Once a permanent change to either an EQUELLA configuration option or the overall architecture is in effect, the previous baselines are no longer relevant and new baselines must be established. performance against an existing baseline. Benchmarking is important because it helps identify changes in performance by comparing against a shard frame of reference. It is important not to vary the execution of tests when performing benchmark tests from the baseline tests, otherwise there is no shared frame of reference with which to compare. Benchmarking should only be used to evaluate differences in hardware, architecture, upgrades and configuration changes. When benchmarking to determine the efficacy of changes, only change one variable at a time. Benchmarking is the process of comparing a system s 9 Conclusion Performance testing is an important step when implementing EQUELLA in the enterprise. When correctly implemented, performance testing helps organisations plan, configure and scale their EQUELLA implementations in their enterprises. Designing performance tests requires asking many questions about how different stakeholders will use EQUELLA. Performance testing in the pilot phase of implementation provides valuable information about how to configure collections and contributions. Since performance testing is an iterative process, it should be expanded with every stage of an EQUELLA rollout. It will become an invaluable tool to assessing whether an organisation s needs are being met. Effective and routine performance testing provides empirical data with which to plan for future growth by measuring the remaining capacity of an existing implementation. Performance Testing I 7
For more information please contact info@equella.com and visit www.equella.com