TESTING AND OPTIMIZING WEB APPLICATION S PERFORMANCE AQA CASE STUDY
2 Intro to Load Testing Copyright 2009 TEST4LOAD Software Load Test Experts
What is Load Testing? Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently. As such, this testing is most relevant for multi-user systems, often one built using a client/server model, such as web applications. Copyright 2009 TEST4LOAD Software Load Test Experts 3
Why Load Test Your Application? High cost of mission-critical application failure Simulate real-life workload conditions Find boundaries of your system Locate potential problems before your customers do Reduce development cycle and time-to-market Predict and reduce infrastructure costs Copyright 2009 TEST4LOAD Software Load Test Experts 4
Load Testing Goals Performance Ensure your application s performance under basic workload (baseline) Stress Find break points and boundaries of your application Reliability How stable is your application under extended period of load Scalability Determine how your application scales for the workload growth Copyright 2009 TEST4LOAD Software Load Test Experts 5
Load Testing and Product Life Cycle Requirements Design Development Testing Production Define performance requirements Design your system based on performance requirements Load test separate components Load test a whole system Monitor your system s performance Copyright 2009 TEST4LOAD Software Load Test Experts 6
Load Testing Workflow Plan Script Test Analyze Improve Copyright 2009 TEST4LOAD Software Load Test Experts 7
8 AQA Case Study Copyright 2009 TEST4LOAD Software Load Test Experts
Background AQA The Assessment and Qualifications Alliance (AQA) is the largest of the three English exam boards. Awarding 49% of full course GCSEs and 42% of A- levels nationally. Processing over 3.5 million exams each year. Providing online services to teachers, exams officers, examiners and students. Copyright 2009 TEST4LOAD Software Load Test Experts 9
SUT ERA (Enhanced Results Analysis) ERA is a free on-line tool that gives teachers in UK an instant analysis of exam results. ERA s key features: Grades reports by centre and subject Cross-centre and years results comparison Individual students grades analysis Question by question analysis Copyright 2009 TEST4LOAD Software Load Test Experts 10
The Challenge Database size over 100 million records 10 million active records 70,000 users 3000 2500 2000 1500 1000 500 0 22/08/2007 - Requests per 10 minutes 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 14/08/2007 - Requests per 10 minutes Copyright 2009 TEST4LOAD Software Load Test Experts 11 3000 2500 2000 1500 1000 500 0 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Workload of less than 10 pages/min at usual day Pick of 300 pages/min at results day
Deployment Architecture Copyright 2009 TEST4LOAD Software Load Test Experts 12
Test Architecture Server Oracle Web Cache Oracle AS Oracle RDBMS Configuration Microsoft Windows Server 32bit Oracle Web Cache 10.2.2 RedHat Linux ES4 U6 Oracle Application Server 10.3.3 ChartFX 6.2 RedHat Linux AS4 U7 Oracle RDBMS 10.2.0.3 Copyright 2009 TEST4LOAD Software Load Test Experts 13
Identifying Test Scenarios 14 Pick 5 most common scenarios in your system Don t add scenarios which operate similar back-end flows Identify problematic back-end flows ahead and make sure scenarios cover them. Copyright 2009 TEST4LOAD Software Load Test Experts
Identifying Potential Bottlenecks Search Usually sensitive to various input values. Simple or not implemented paging can cause to slow response time Search Results If paging is not implemented, result page can be very large Back to search results page from item details can execute search again External and 3rd Party Components 3rd party components can cause to deadlocks or poor performance OSSO server can perform slower than the rest of the system Export to Excel Usually returns large result set. Can cause to slow performance. Copyright 2009 TEST4LOAD Software Load Test Experts 15
Execution Plan Test the system with light input parameters first Get through the obvious bottlenecks, constantly increasing the workload Run nightly tests every night to ensure reliability and stability of the system At level of 500 users, apply heavy input parameters Test system for randomly distributed workload Copyright 2009 TEST4LOAD Software Load Test Experts 16
10 Users Conn. Pool Starvation Measurement Avg./Sum Max. Page Time 1.6 sec 34 sec HTTP 200 3,348 HTTP 304 2,190 HTTP 302 410 HTTP 500 48 100% 80% 60% 40% 20% 0% Connection Pool Connections Copyright 2009 TEST4LOAD Software Load Test Experts 17
100 Users Apache Shows Its Limit Measurement Avg./Sum Max. Load Size 100 Page Time 5.6 sec 67 sec Pages per Second 2.1 7.5 HTTP 503 8 Copyright 2009 TEST4LOAD Software Load Test Experts 18
500 Users Slow DB & OSSO Measurement Avg/Sum Max. Load Size 1000 Page Time 24 sec 2.5 min Pages per Second 7 19 HTTP 500 368 HTTP 503 157 Copyright 2009 TEST4LOAD Software Load Test Experts 19
1000 Users Search & Large Content Measurement Average/ Sum Max. Load Size 1000 Page Time 1 min 7.5 min HTTP 500 27 HTTP 503 626 Copyright 2009 TEST4LOAD Software Load Test Experts 20
Concluding 1000 Users 8 hrs Test Measurement Average/ Sum Max. Load Size 1000 Page Time 0.717 sec 6.169 sec Copyright 2009 TEST4LOAD Software Load Test Experts 21
Load Issues Pyramid 10 Conn. Starvation 100 Apache Limit 500 Slow DB Queries 1000 Code, Search, Cache Copyright 2009 TEST4LOAD Software Load Test Experts 22
Achieved Results Supported concurrent sessions/users: From 10 to 1000 (x100) Reliability From 90% failure to ~100% reliable system Response Time at 1000 VU From 1 min to less than a second Pages per Minute From 13 PPM to 166 PPM Copyright 2009 TEST4LOAD Software Load Test Experts 23
24 Tips for Effective Load Testing Copyright 2009 TEST4LOAD Software Load Test Experts
Optimizing Search Mechanism Use existing solutions. Use free text search only as the last option. Limit user input to reasonable values. Implement paging on both search and result levels. Improve Back to search results mechanism. Copyright 2009 TEST4LOAD Software Load Test Experts 25
GZip Compression Effectiveness Pros: Reduce size of large text pages by 80%-90% Improve response time by 50% Decrease throughput Cons: Increased CPU utilization by 35% Tip: Apply GZip selectively to text-based static resources and large dynamic pages Do not apply GZip to every response! Copyright 2009 TEST4LOAD Software Load Test Experts 26
Database vs Web Servers Costs Additional database setup and maintenance costs are significantly higher than of web server. Make DB work easier by making web server work harder. Memory costs less than CPU or new hardware rely more on memory, but keep in mind that everything has its limits. Copyright 2009 TEST4LOAD Software Load Test Experts 27
Developers and Testers Tips Get to know your user. Evaluate his system usage behavior or try to predict the common flow. Know the costly operations of your system. Try to build the flow that omits recurrent usage of the operations. Do not be reluctant to cache data, still don t overuse the cache. Remember that shared data should be stored once for all users. Copyright 2009 TEST4LOAD Software Load Test Experts 28
Project Manager Tips Make sure all involved parties are available at the moment of test execution. Share test findings among all project members. Log all changes to the test environment. You will need to reapply them later to the production environment. Make sure the final results meet the performance requirements and business user expectations. Copyright 2009 TEST4LOAD Software Load Test Experts 29
Team Synergy Synergy and cooperation between the following teams is crucial for the success of the test: Database Team Application/Development Team System Team Performance Team or Specialist Copyright 2009 TEST4LOAD Software Load Test Experts 30
Questions? Contact Info Nachum (Nick) Dimer Owner & Chief Consultant IL: +972.54.4600515 UK: +44.783.8673770 PL: +48.519.717875 ndimer@test4load.com www.test4load.com