TRUE PERFORMANCE ENGINEERING

Transcription

1 TRUE PERFORMANCE ENGINEERING Quality with Testing, Testing with Quality WHITE PAPER TWO

2 CONTENTS Introduction 3 The Challenges 3 Dependable Applications Testing 3 Traditional Load Testing 4 Large Capital Expenditure 4 Incomplete Applications Testing 4 One-shot Performance Test scripts 4 Complex Browser testing 4 Right information at the same time 4 Diagram 1 Application Overview 5 The Vision 5 The Solution 5 Diagram 2 Approaches to Performance Engineering 6 Minimal Capital Expenditure 6 Re-useable Test scripts 6 Complex Browser testing 7 True Picture of Real-User Experience 7 Rapid and low cost availability 7 Diagram 3 Load Management 7 Key benefits 8 Diagram 4 Load Injector Platform 8 Conclusion 9 2 TTP TRUE PERFORMANCE ENGINEERING

3 INTRODUCTION Complex applications leveraging real-time interactive push services are becoming the norm. The rate at which these services present streamed dynamic push data, presents a new scale of opportunity for providers to offer differentiated services where clients subscribe to applications with constantly changing price screens, performance indicators, 3d graphics and analytics. User interaction and ongoing loyalty depends on the accuracy, speed and reliability of the data presented, and as applications wait for the user to commit to bet, buy or sell or place an order massive volumes of data are being simultaneously transported across the web. As a consequence of the requirement to push data such as prices, gaming odds and other real time data, accurately and identically to all parties, true performance engineering, quality performance and stress testing are becoming more and more critical. Traditional testing tools, which remain effective for static data functions in mainframe and pull data applications as seen in many retail websites, are proving to be inadequate for today s complex rich internet applications where complexity of functionality within the client is increasing, this is particularly the case in systems where the volumes of pushed data to the client is driving markets and profitability. This white paper explores the traditional versus new methodology of performance testing, and explains how The Test People s solutions mitigate risks by enabling high volume, true performance engineering and load testing, with full visibility of speed, flows and access to simultaneous pricing. This paper is intended for CIOs, IT directors and managers, in enterprises of all sizes, especially those who have the need to test evolving complex client applications with true performance engineering in the Cloud. THE CHALLENGES Dependable Applications Testing Credibility and dependency on real time data is becoming paramount across a range of industries, as customers exercise choice of providers, it is essential that providers are able to demonstrate the highest levels of true performance to retain clients. Users will change provider if screens do not appear to be updated in real-time, or if confirmations are requested for prices or goods, which, due to market or stock movement are then unavailable. The evolving nature of business critical systems has lead to demands for more tests to cover functionality and operational detail, new information distribution methods and interoperability of applications with browsers and platforms. Testing is now much more complex as there are now many different browsers, delivery platforms and protocols that must be taken into account. Consequently, the time, cost and effort to test these inter-dependencies, has become much greater, and in some cases unmanageable. Other issues such as slow screen rendering or perceived system slow down will also affect user confidence, so providers must have some assurance that their components are running correctly end-to-end and they are able to enhance user experience. There are so many complex interplays and elements to modern systems that identifying where bottlenecks are, and how to build in quality to ensure customers stay with one provider are key priorities. Performance, application and volume testing is vital, but traditional testing tools have limitations, as highlighted below. 3 TTP TRUE PERFORMANCE ENGINEERING

4 TRADITIONAL LOAD TESTING Testing of application loading typically involved replicating the traffic between client and server and checking the time for messages to be returned. Replicating the user activity in this way proves that the server is performing. While this is very important, it does not give true visibility as to how fast data is being returned to users by the client, and whether this data is distributed consistently, securely and accurately. LARGE CAPITAL EXPENDITURE Building the necessary hardware for internal injector farms (racks of servers and communications infrastructure), the costs of an internal QA lab, test software, test maintenance and expert consultants may be cost-prohibitive for many organizations who need to scale to potentially millions of users. Also, these projects tend to be inefficient, as servers need to be provisioned and scaled for the worst-case load tests and are redundant when testing projects are not taking place. Additionally, budget constraints may mean that insufficient hardware can be purchased to replicate maximum loads. INCOMPLETE APPLICATIONS TESTING Capturing and maintaining a large user base on the Internet is dependent on fast, reliable, low latency accurate content and delivery. The complexity of modern applications means that content comes from multiple sources, mixing external static or dynamic data with their own internal information. Organisations may use their own injector farms, however the challenge is compounded in that they may not fully replicate an environment with its rapidly changing mix of internal and external sources and connectivity inter-dependencies. ONE-SHOT PERFORMANCE TEST SCRIPTS The dynamic nature of data and integrated applications means that every time content, volume or versions change, new test scenarios need to be created to test environments and the datasets within them. COMPLEX BROWSER TESTING As more application functionality is executed at the client, the use of traditional performance testing methodologies to simulate message-level interaction with the server for Rich Internet Applications (RIAs) is becoming more complex and potentially less representative of the behaviour of the application itself. Many Modern web applications now depend on the use of application frameworks such as SilverLight, Flash and JavaFX or ever increasingly complex javascript, and their compatibility with browsers such as IE, Firefox or Chrome. RIGHT INFORMATION AT THE SAME TIME Typically, large volume systems deliver information to tens of thousands of subscribers simultaneously, each with their own preferences and licenses, and the vendors internal systems must gather the prices, and send subscribed data to each client. However as prices change, there is often no way to know the true fan-out of this information, how long updates have taken to reach the user, or how long the browser may take to parse the information, potentially leaving users blind to latest prices and the vendor incorrectly believing they have fulfilled their obligations in supplying on-time service. For example, this is true in Foreign Exchange markets, where internet-based individuals are empowered to trade in real-time, waiting to exercise their buy or sell strategies through a plethora of browsers. In fast moving markets, users with complex trading applications need to be assured that they are receiving prices and trading opportunities accurately and quickly as they scan the constant stream of activity. For an e-gaming or financial platform provider, the key is to be able to broadcast fast and accurate data, with proven indicators that all users are operating in a transparent market, without disadvantaging any clients. Testing these environments presents new levels of challenge, as the scope, cost and resource utilisation may be enormous as firms would need access to massive amounts of computing power to run meaningful tests. It is possible to fool a server into accepting scripts replicating the client server traffic from a user on an internal injector farm. However, to simulate scale and run a true multi-user test, involving re-rendering of real-time screens on thousands of user desktops is an enormous overhead involving many servers, time, resource and expense. 4 TTP TRUE PERFORMANCE ENGINEERING

5 DIAGRAM 1 APPLICATION OVERVIEW A User Click Button Enter Text B Client Browser processing HTTP Protocol variants: AMF,RTMP,XML,STOMP Or Bespoke Binary Formats C Datacentre Read Back Values TCP / IP IN PUSHED DATA APPLICATIONS: Interaction between users and the (A to B) is relatively simple, the user subscribes to a number of data channels, data is returned automatically when changes occur and the user then acts on the data when they wish, for example, placing bets and placing/modifying/cancelling order and trades. Interaction between the and the server (B to C) is complex. The may use differing protocols to communicate with the server negotiated at connection time depending on network capability. When updates are received from the it executes business logic on the data updating its own internal data structures, it may autonomously query the server or another 3rd party for additional information and then render the updates on the screen to the user. All of this happens automatically within the without any interaction between the user and the. THE VISION To remain competitive companies must engage optimal true performance engineering methodologies to ensure they are fully aware of and are able to manage any latency and potential bottleneck issues with their system. Competitive advantage may also be gained as a consequence of being able to demonstrate to clients that optimal levels of performance are consistently achieved. The vision therefore is to provide service providers with cost effective and easy to deploy performance tools, environments and analytics such that are able to prove consistently superior levels of performance and service delivery. THE SOLUTION As evidenced by industry experts and client experience, performance testing of pushed data systems is not achievable using traditional performance engineering methods. Consequently, a new approach is required. The True Performance Engineering approach has clear advantages. It enables handling of dynamic data, multiple applications, complex hardware (injector farm capability), software dependencies and huge volumes. It also allows better measurement of budget, risk, resource scalability and availability. 5 TTP TRUE PERFORMANCE ENGINEERING

6 DIAGRAM 2 APPROACHES TO PERFORMANCE ENGINEERING Approach Traditional Approach Controller Server Controller Real Protocol Level Comms Protocol Level Comms Simulated Client User Actions Real Simulated Client User Actions Real Simulated Client Protocol Level Comms Protocol Level Comms Uses the REAL Controls the through User Interaction Observable responses are when the user sees a response Complexity is abstracted from the load tester Regardless of complexity or type of protocol used, interaction is performed at user level (key presses, mouse clicks etc.) Does NOT use the real, but mimics the protocol level messaging to SIMULATE the behaviour. Rework test scripts if protocol activity changes Monitor protocol level response times if the is adding to response times this will not be picked up business logic has to be replicated through performance test tool scripts Tool vendors must provide support for increasing numbers of dynamic, differing protocols MINIMAL CAPITAL EXPENDITURE True performance engineering traditionally demanded massive infrastructure and resource utilisation at substantial cost, however, the cloud now makes it possible to access immense, on demand, cost-effective, scalable cloud based injector farms, leased solely for the duration of the test. RE-USEABLE TEST SCRIPTS Tests scripts written to exercise the client are more robust and require less maintenance than traditional load test scripts simulating client and browser interaction. 6 TTP TRUE PERFORMANCE ENGINEERING

7 COMPLEX BROWSER TESTING Using frameworks and applications within the Cloud, based on virtual servers with wire capture capability, there is full understanding of the protocols involved, and how data is moved and accessed specifically for the tests. The Test People have deployed injector models enabling running of many thousands of clients in truly representative scenarios driving the real web browser (or windows based client), with an in-depth appreciation of the technology and cost implications. TRUE PICTURE OF REAL-USER EXPERIENCE In protocol level performance testing the transaction time clock stops when the simulated client receives a response back from the server. From an end user perspective however the transaction is only completed after the client has received the message, processed the message and rendered the update on the screen. The Test People have used their true performance engineering and smart client application testing experience to performance test the client itself and include client processing time in reported transaction times, accurately capturing the elusive end-to-end response time as observed by real users. This measure is becoming increasingly important in an environment where bottlenecks or performance issues in a complex browser application can severely impact end-user transaction times and go unnoticed in protocol level testing. RAPID AND LOW COST AVAILABILITY Cloud computing offers pay for what you use operations, enabling on demand expansion of load infrastructures without the need for scheduled load management, often months in advance. In addition to the technical advantages, this model offers end-users true business benefits in terms of flexible project prioritisation and reductions in hardware budgets, which directly affect bottom line return on investment. The TTP solution enables clients to gain significantly more confidence in their production systems. With experience of providing true performance engineering, testing skills and architectural advice, industrial strength is proven faster, with improved application quality and speed of release cycles. By harnessing the cloud, the ability to cost-effectively manage and monitor exponential load tests has finally arrived. DIAGRAM 3 LOAD MANAGEMENT System Under Test Load Injector 1 Notes Load Management server orchestrates all data to/from ALL load injectors Load injector comprises a Load Orchestration Module (applies the user journeys to the s) Load Orchestration Wire - Capture/Decoder Wire Capture/Decoder sniffs messages from the wire, decodes data and sends data back to the Load management server Load Management Server Load Injector 2 Deserialised Updates Load Injector N 7 TTP TRUE PERFORMANCE ENGINEERING

8 KEY BENEFITS Significantly faster scripting allows delivery of performance testing results more quickly Fast performance testing of browser/applications combinations I/O of messages can be monitored and managed through their life-cycle Testing scenarios are scalable, less brittle and not reliant on underlying protocols. Cost efficiency - only using the cloud resources as required for limited periods No interference with live operations Promotes rapid deployment of new solutions Still gives visibility of underlying messaging between client and server. With proven industrial strength load injector tools enabling providers to test their applications in the process of broadcasting push prices accurately and speedily, the Cloud based true performance engineering solutions of The Test People offer flexible on demand virtual unlimited load capability at low cost whilst also bringing other benefits such as more robust and easier to maintain performance test scripts. DIAGRAM 4 - LOAD INJECTOR PLATFORM Injector 1 Control Load Orchestration Wire - Capture/Decoder Protocol Level I n System Under Test Load Management Server Control Stats / Prices Stats / Prices Load Orchestration Injector 2 Wire - Capture/Decoder Protocol Level t e r n e t Stats / Prices Control Load Orchestration Injector N Wire - Capture/Decoder Protocol Level Notes Many load injections talking to a single load management server 8 TTP TRUE PERFORMANCE ENGINEERING

9 CONCLUSION True Performance Engineering enabling high-volume, data-intensive push applications to obtain meaningful information on stability, interoperability and scalability is vital. As such, True Performance Engineering in the cloud has emerged as a cost-effective, secure and resilient solution to testers and developers by delivering bandwidth and computing power to run real-time exact replica tests at user levels never before possible, enabling industry strength applications to be delivered with reliability, certainty and time critical accuracy. The problems that arise from untested and incorrectly tested complex real-time push client applications can result in loss of customers, revenue and credibility, but powerful, cost-effective testing solutions are available to protect your business. With its experience in multiple methods and markets, The Test People are leaders in this specialist field, where understanding the issues from a content delivery, RIAs and end user perspective has enabled it to identify and monitor fan-out at the client - the all important time between the first and last user update for a single piece of information, the implications of which may affect customer retention, loyalty and trust. The Test People s suite of solutions give users peace of mind that testing truly reflects the scale and variety of actual usage, delivering a framework for push platform providers and ultimately the customers to interact with what they need, reliably, with low latency, simultaneously, at the time they need it. For corporations seeking to seamlessly market and move their data across multiple digital channels and empower push applications, with advanced delivery capabilities, comprehensive monitoring, analysis, powerful management and true performance engineering tools, The Test People provides a compelling combination. To learn more about true performance engineering in the cloud, visit for more details. CONTACT THE TEST PEOPLE Phone: Leeds Office +44 (0) contactus@thetestpeople.com London Office: +44 (0) ABOUT THE TEST PEOPLE: Founded in 2007 and with offices in Leeds and London, The Test People Ltd. Is an innovative testing consulting firm specializing in True Performance Engineering, Test Automation and Test Management. Our service is designed to provide the best Testing and True Performance Engineering Services possible to either underpin an existing Test Team or Strategy and can be tailored especially to your business or a particular project. 9 TTP TRUE PERFORMANCE ENGINEERING