G-Cloud Service Definition. Performance Testing as a Service (PTaaS) SCS

Transcription

1 G-Cloud Service Definition Performance Testing as a Service (PTaaS) SCS

2 Atos Performance Testing as a Service (PTaaS) SCS PTaaS is performance testing from the cloud, on demand with minimal capital investment (see diagram below). Suitable for most types of client, it is a highly-scalable, low-cost service, without complex contracts and lengthy lead times. Atos Performance Testing as a Service (PTaaS) A low-cost entry into Performance Testing Instead of significant investment on performance tooling licences and building in-house performance test infrastructure, the Atos customer pays for an external service, PTaaS. Also, because it s a service Atos customers don t have to involve so many of their busy IT staff, delivering a further cost reduction Lower risk investment Atos customer pays for PTaaS on demand, as discrete Units of Work (UoWs). The monetary risk is lower, and amortised over time, as it moves the cost from capital to operational expenditure Value for money Atos customer no longer has Capex and the fixed costs of maintaining the tools, the team and the infrastructure to generate the performance load. Instead Atos customer pays only for performance test capacity they want to use. The Atos customer has the freedom to order performance test capacity only when needed Huge scalability and reduced delivery times To emulate real-world usage of Atos customer s website application, there is no practical limit to the number of Virtual Users (VUs) that can be generated by Atos cloud-based load injectors. Furthermore, Atos cloud-based infrastructure can be provisioned on demand in a fraction of the time it would take to procure and implement new, in-house load injectors. Costs are proportionately higher with greater numbers of VUs but as Atos customer pays for a service rather than for permanent licences, the costs scale economically Enhanced capacity planning Together with Atos expert resources, PTaaS enables Atos customer to make informed decisions for timely capacity planning and procurement, processing of more transactions and more efficient use of existing capacity. PTaaS test tools are hosted in the cloud and can generate more realistic production-like load scenarios from geographically distributed injection points. PTaaS allows Atos customers to inject global production loads into their system under test. Allowing them to discover how their systems behave at high load before their users do. Atos is committed to working with its customers to aid their understanding of how PTaaS is best employed to plan and execute projects that harness new technologies to deliver competitive business advantages. ii

3 What is it? In short, Atos customers can realistically model exactly what will happen to their web application before they go live. This allows them to address performance deficiencies early in a measured and planned way, rather than under the expensive spotlight of disgruntled users and disenfranchised stakeholders. Atos is committed to working with its customers to aid their understanding of how PTaaS is best employed to plan and execute projects that harness new technologies to deliver competitive business advantages. Atos clients are now benefiting from considerable savings through the adoption of Atos Performance Testing as a Service. Most organisations could save at least 25% on their performance test costs. A government agency Atos worked with recently saved 100,000 over traditional tool procurement and permanent tool hosting, budgeted to cost 250,000. Atos will be very pleased to work with future customers to achieve similar savings. What makes us unique? The service is a very flexible and secure, and can be used in many different situations. For example: For testing of Non-functional requirements; Volume, Load, Response, Capacity and Stability / Soak, Scalability To assess and tune current infrastructure capacity; Assessing virtualised platforms (VMWare ESX, IaaS, PaaS),using a variety of different usage models on a hosted application Backend performance testing; E.g. Testing of messaging buses (Oracle Service Bus OSB, TIBCO, JBoss ESB, Redhat FUSE and IBM Websphere to list a few), Web services & WSDL interfaces, Database performance and other custom backend platforms Frontend testing; e.g. testing of standard HTTP web frontends (including AJAX), application services, Siebel, SAP and standardised web technologies Tuning configuration; Performance engineering and tuning of JVMs, Connection Pooling, Load-balancing infrastructure, software frameworks and database configuration. Atos PTaaS is delivered from a set of Units of Work (UoW), each with defined inputs, outputs and benefits UoWs are modular and assemble flexibly to provide complete performance testing solutions Atos PTaaS is Software Development Life Cycle (SDLC) agnostic and applicable to Waterfall, V Model and Agile methodologies. The UoW can be applied early in the SDLC to mitigate performance and load risks where the remedial cost is low. The core Units of Work are: Performance Test Assessment - Assesses the customer s readiness for PTaaS and identifies any known performance trouble spots and risks. Proof of Concept (PoC) - Proves the tool against the system under test regarding technology and access iii

4 Key Transactions Performance Tests - Identify key business transactions then build and execute defined set of performance scripts. Report on results against the agreed key performance indicators. Full Analysis, Diagnosis and Recommendations - Root cause analysis and recommendations for further performance tests to investigate in more depth. iv

5 v

6 Contents 1. Introduction Service summary How this product can be used Service overview Four Ways that PTaaS can boost Delivery: Helping Atos Clients to make Cost Savings Service Roadmap Information assurance Backup/restore and disaster recovery On-boarding and off-boarding On-boarding Off-boarding Pricing Service management Service constraints Service levels Financial recompense Training Ordering and invoicing process Termination terms By consumers (i.e. consumption) By the Supplier (removal of the G-Cloud Service) Data restoration / service migration Consumer responsibilities Technical requirements Trial service vi

7 1. Introduction Every business wants to achieve high performance from their systems; few realise this without significant investment in time and resources. The best, but often prohibitively expensive, solution to performance-related risks is appropriately implemented automated performance testing. However, few can justify the eye-watering investment required to build, maintain and staff an in-house performance test capability. The majority, of course, cannot do so, and have had to gamble that their web-based applications: Will be able to handle peak volumes of users and data Will recover from high-water marks of extreme user activity, and Will be able to run over lengthy periods without degraded performance leading to system outages. These organisations have had to endure painfully anxious, white-knuckle rides into production; their only defence to performance failures being to react rapidly in an effort to minimise downtime, reputational damage and financial losses. What Atos customers have been demanding is a more economical, flexible and faster way to undertake performance testing, expediting the launches of new web applications, confident in the knowledge that they will be high performing, robust and fit for purpose. 1.1 Service summary In response to the needs of Atos customer s challenges, we ve designed Atos Performance Test as a Service (PTaaS). PTaaS Leverages Atos wealth of performance test experience with large-scale commercial and central government systems, PTaaS provides: A low-cost entry into Performance Testing. Instead of significant investment on performance tooling licences and building in-house performance test infrastructure, Atos customer pays for an external service, PTaaS. Also, because it s a service Atos customers don t have to involve so many of their busy IT staff, delivering a further cost reduction Lower risk investment. Atos customer pays for PTaaS on demand, as discrete Units of Work (UoWs). The monetary risk is lower, and smoothed over time, as it moves the cost from capital to operational expenditure Value for money. Atos customer no longer has Capex and the fixed costs of maintaining the tools, the team and the infrastructure to generate the performance load. Instead Atos customer pays only for performance test capacity they want to use. Atos customer has the freedom to order performance test capacity only when needed Huge scalability and reduced delivery times. To emulate real-world usage of Atos customer s website application, there is no practical limit to the number of Virtual Users (VUs) that can be generated by Atos cloud-based load injectors. Furthermore, Atos cloud-based infrastructure can be provisioned on demand in a fraction of the time it would take to procure and implement new, in-house load injectors. Costs are proportionately higher with greater numbers of VUs but as Atos customer pays for a service rather than for permanent licences, the costs scale economically 1

8 Enhanced capacity planning. Together with Atos expert resources, PTaaS enables Atos customer to make informed decisions for timely capacity planning and procurement, processing of more transactions and more efficient use of existing capacity. 1.2 How this product can be used Atos Performance Testing as a Service provides performance & load testing ondemand and via the cloud. The service is a very flexible and secure, and can be used in many different situations. For example: For testing of Non-functional requirements; Volume, Load, Response, Capacity and Stability / Soak, Scalability To assess and tune current infrastructure capacity; Assessing virtualised platforms (VMWare ESX, IaaS, PaaS),using a variety of different usage models on a hosted application Backend performance testing; E.g. Testing of messaging buses (Oracle Service Bus OSB, TIBCO, JBoss ESB, Redhat FUSE and IBM Websphere to list a few), Web services & WSDL interfaces, Database performance and other custom backend platforms Frontend testing; e.g. testing of standard HTTP web frontends (including AJAX), application services, Siebel, SAP and standardised web technologies Tuning configuration; Performance engineering and tuning of JVMs, Connection Pooling, Load-balancing infrastructure, software frameworks and database configuration. 2

9 2. Service overview PTaaS is performance testing from the cloud, on demand with minimal capital investment (see diagram below). Suitable for most types of client, it is a highlyscalable, low-cost service, without complex contracts and lengthy lead times. PTaaS test tools are hosted in the cloud and can generate more realistic production-like load scenarios from geographically distributed injection points. PTaaS allows Atos customers to inject global production loads into their system under test. Allowing them to discover how their systems behave at high load before their users do. Atos is committed to working with its customers to aid their understanding of how PTaaS is best employed to plan and execute projects that harness new technologies to deliver competitive business advantages. 2.1 Four Ways that PTaaS can boost Delivery: Efficiency PTaaS delivers a raft of efficiencies leading to a reduction in overall IT costs: It is quick to start up It is massively scalable within short timescales It is charged on a per-use basis; specific UoWs are delivered as needs dictate It does not require capital investment in test tool licenses or hardware, so the entry cost is lower and money is not wasted when performance tests are not being run Risk reduction PTaaS reduces risk with its unique UoWs approach: UoWs break the work into smaller, more manageable chunks each delivering defined benefits UoWs facilitate early and regular progress tracking within the performance testing life cycle, allowing verification that the service is meeting its needs 3

10 Specialists Choose PTaaS and leave it to us: Atos customer is in the safe hands of Atos Subject Matter Experts who will deliver performance solutions in UoWs to ensure Atos customers business succeeds Atos experts are experienced in performance testing across all technologies to very high volumes, effectively harnessing the capabilities of the cloud Transparency PTaaS lets its customers see where their money is going and budget use of the service accordingly: UoWs are priced individually so Atos customers can pick and choose which are applicable to their situation Opex model makes it easy to track the cost of Atos customers performance testing Easy to relate business benefits to specific UoW, allowing focus to be placed on performance test activities that deliver the greatest value. 2.2 Helping Atos Clients to make Cost Savings PTaaS is performance testing from the cloud, on demand with minimal investment. Suitable for most types of client, it is a highly-scalable, cost efficient service, without complex contracts and lengthy lead times. Because PTaaS test tools are hosted in the cloud more realistic production-like load scenarios can be generated from geographically distributed points of injection. In short, Atos customers can realistically model exactly what will happen to their web application before they go live. This allows them to address performance deficiencies early in a measured and planned way, rather than under the expensive spotlight of disgruntled users and disenfranchised stakeholders. Atos is committed to working with its customers to aid their understanding of how PTaaS is best employed to plan and execute projects that harness new technologies to deliver competitive business advantages. Atos clients are now benefiting from considerable savings through the adoption of Atos Performance Testing as a Service. Most organisations could save at least 25% on their performance test costs. A government agency Atos worked with recently saved 100,000 over traditional tool procurement and permanent tool hosting, budgeted to cost 250,000. Atos will be very pleased to work with future customers to achieve similar savings. Quick tour of the Units of Work Atos flexible, modular framework for delivering PTaaS is illustrated below: 4

11 Atos PTaaS is delivered from a set of UoWs, each with defined inputs, outputs and benefits UoWs are modular and assemble flexibly to provide complete performance testing solutions Atos PTaaS is Software Development Life Cycle (SDLC) agnostic and applicable to Waterfall and Iterative methodologies The UoW can be applied early in the SDLC to mitigate performance and load risks where the remedial cost is low. The core Units of Work are: Unit of Work (UoW) Performance Test Assessment Proof of Concept (PoC) Key Transactions Performance Tests Full Analysis, Diagnosis and Recommendations Summary Assesses the customer s readiness for PTaaS and identifies any known performance trouble spots and risks. Proves the tool against the system under test regarding technology and access Identify key business transactions then build and execute defined set of performance scripts. Report on results against the agreed key performance indicators. Root cause analysis and recommendations for further performance tests to investigate in more depth. 2.3 Service Roadmap Atos use a wide range of performance test products and which are constantly evolving to keep pace with technology. The use of Atos cloud based test tools means Atos customers get the latest versions without waiting for them to be installed on in-house environments. 5

12 3. Information assurance This product is currently available at Impact Level 0 (IL0). The service can be run at higher Impact Levels including IL2 and IL3. Atos has considerable experience of providing services at different levels of assurance. Atos currently has a number of products on G-Cloud that have received Pan Government Accreditation (PGA). Details can be found on the Cabinet Office website at: 6

13 4. Backup/restore and disaster recovery Normal backup/restore and disaster recovery arrangements are in place on Atosowned infrastructure. Where client-owned infrastructure is used, Atos usually rely on the client s own arrangements but can quote for special arrangements. 7

14 5. On-boarding and off-boarding Integration of Atos Performance Testing as a Service (PTaaS) into an existing project has been designed to be as simple as possible. Atos follow Atos UoW (Unit of Work) framework. 5.1 On-boarding Installation takes less than a day, though Atos recommend scheduling a day to be on the safe side. In advance of day one, Atos will provide a Pre-requisites document. This document outlines the steps and requirements that will be needed to have in place. This will include, for example, Business Requirements (including Nonfunctional requirements: Volumes, Usage Models etc.), Environment constraints and connectivity, required technologies and scope of testing. On the first day, Atos trained performance engineers will set about checking connectivity and starting the process of getting up to speed on the target system(s). 5.2 Off-boarding On completion of the PTaaS activities, Atos will deliver the test scripts, logs and raw test results via a preferred and secure data medium. The formal test reports will have been delivered in the UoW Key Transactions Performance Test. N.B The test scripts will be in the format only readable by the performance test tool, however these will be reusable again for future assignments. 8

15 6. Pricing The table below is the daily chargeable rate per consultant. (This includes the use of the tool during the design stage only see the table below for execution) Please refer to the SFIA rate card - Atos (On & Offshore) for the standard rates for this service. The primary roles which are engaged in delivering this service are: Service item SFIA Units/Notes Performance Test Analyst 4 per person per day Performance Test Lead 5 per person per day Performance Test Consultant 7 per person per day The table below is the packages daily chargeable rate of the tool during execution using Neoload Performance Tooling supplied by Neotys. (see Section 2). Execution effort and duration will be defined during the Performance Test Assessment UoW. PT Execution Package #Users Cost Execution for 1 protocol day Additional Cost per protocol/day Small (Worldwide Cloud Injectors, 1 Load Generator) Medium (Worldwide Cloud Injectors, 2 Load Generators) Large (Worldwide Cloud Injectors, 5 Load Generators) Small (Onsite tooling) Medium (Onsite tooling) Large (Onsite tooling) Further discounts for more than 2500 users are available. 9

16 To illustrate how the costs are constructed: Worked example 1: Project of 20 days Target Volume 300 Vusers. Web/HTTP & Siebel Basic Monitoring 15 days of Requirements and Design work 5 working days of Execution (the smallest chargeable value is one day) (20 days * Performance Test 960) + (5 days * Small PT 500) ( 19,200) + ( 2,500) = 21,700 Worked example 2: Project of 30 days Target Volume 1500 Vusers. Web/HTTP & Siebel & Silverlight Basic Monitoring 20 days of Requirements and Design work 10 working days of Execution (the smallest chargeable value is one day) (30 days * Performance Test 960) + (10 days * large PT 900) + (10 days * 240) ( 28,800) + ( 9,000) + ( 2,400) = 40,200 The costs above are subject to the following conditions; Execution duration is a minimum of 4 days. Execution assumes Web/HTTP + One specialised protocol Specialised protocol: Oracle Forms, Siebel, Java Serialisation, Silverlight, SOAP, GWT, Push, RTMP, Flex Subsequent additionally required protocols are charged according to the additional protocols table above Basic Monitoring is included Specialised Monitoring includes in system monitoring of: App Servers; LCDS, WebSphere, Oracle App Server, WebLogic, Tomcat, JMX, JBoss, JOnAS, GlassFish, NetWeaver SAP,.Net Databases; Oracle Database, DB2, PostgreSQL, SQL Server, MySQL Network; SNMP, RSTAT OS; HP-UX, Linux, AIX, Solaris, VMWare, Windows Web Servers; IIS, Apache. 10

17 7. Service management Atos services normally operate within an overall project management context. Where this service is purchased free-standing, a project/service management wrapper will be built in to work plans and effort estimates. This will normally include status/progress reporting. 11

18 8. Service constraints The service may be constrained by: The bandwidth and latency of the network access to the system under test The capability of the infrastructure hosting the system under test Access to the system under test from the internet for which Atos will need to consider non-cloud load injection options Impact assessment of the consequences of running production like loads on the other IT services sharing the same IT infrastructure as the system under test. 12

19 9. Service levels Atos provides suitably trained individuals to complete the tasks necessary for this service. They can work at the client site or remotely, depending on the need for access to: Client staff Client applications & network. Standard Working Hours/Days Monday to Friday 09:00 to 17:30 excluding public & regional holidays. 13

20 10. Financial recompense To minimise the cost to users, Atos does not provide service credits for use of the service. All Atos services are provided on a reasonable endeavours basis. Atos provides a free trial service so that customers may experience and experiment with the service in advance of any purchase. This is intended to allow customers an ability to assess whether the service Atos offers meets the customer s needs. In accordance with the guidance within the GPS G-Cloud Framework Terms and Conditions, the Customer may terminate the contract at any time, without cause, by giving at least thirty (30) Working Days prior notice in writing. The Call Off Contract terms and conditions and the Atos terms will define the circumstances where a refund of any pre-paid service charges may be available 14

21 11. Training Test training is not required to order or benefit from this service. Any special test terms or techniques will be explained in the test documentation. 15

22 12. Ordering and invoicing process Ordering this product is a straightforward process. Please forward your requirements to the address GCloud@atos.net Atos will prepare a quotation and agree that quotation with you, including any volume discounts that may be applicable. Once the quotation is agreed, Atos will issue the customer with the necessary documentation (as required by the G-Cloud Framework) and ask for the customer to provide Atos with a purchase order. Once received, the customer services will be configured to the requirements as per the original quotation. For new customers, additional new supplier forms may need to be completed. Invoices will be issued to the customer and Shared Services (quoting the purchase order number) for the services procured. On a monthly basis, Atos will also complete the mandated management information reports to Government Procurement Services detailing the spend that the customer has placed with us. Cabinet Office publish a summary of this monthly management information at: 16

23 13. Termination terms 13.1 By consumers (i.e. consumption) Termination shall be in accordance with: The G-Cloud Framework terms and conditions Any terms agreed within the Call Off Contract under section 10.2 of the Order Form (termination without cause) where the Government Procurement Service (GPS) guidance states At least thirty (30) Working Days in accordance with Clause CO-9.2 of the Call-Off Contract Atos Supplier Terms for this Service as listed on the G-Cloud CloudStore. For this specific service, by default Atos ask for at least thirty (30) Working Days prior written notice of termination as per the guidance within the GPS G-Cloud Framework Terms and Conditions By the Supplier (removal of the G-Cloud Service) Atos commits to continue to provide the service for the duration of the Call Off Contract subject to the terms and conditions of the G-Cloud Framework and Atos Supplier Terms. 17

24 14. Data restoration / service migration Data restoration Atos will restore test data and scripts into the cloud based test tools if it is required. Data restoration for the system under test will be the responsibility of the provider of the system under test. Service migration Upon exit from Atos PTaaS solution Atos deliver the test scripts, logs and raw test results as they exist at the time of migration cut-over. In the event of service migration to another performance test tool then the new performance test provider may need to translate or rewrite the test scripts and the load data files. 18

25 15. Consumer responsibilities The principal consumer responsibilities are: Provide the system under test Provide the system under test with access to the internet of a suitable bandwidth and latency for the tests to be realistically executed. If internet access is not permissible or inadequate then to provide alternative hosting for the performance tools and authorisation for Atos test specialists to load the tooling onto the host provided Provide data preparation and data loading of the system under test prior to test execution Provide support, maintenance, monitoring and tuning of the system under test during the PTaaS engagement Provide data cleansing and data restore after test execution Provide all required authorisation for access to the system under test Gain internal authorisation that the planned performance load can be injected across the consumer s IT infrastructure on the agreed test execution days. 19

26 16. Technical requirements There are technical requirements for integration of the PTaaS infrastructure into Atos customers test environment, however, these depend on the type of infrastructure that is running whether it is in a secure environment and thus cannot be accessed publicly, and finally the types of protocols used on the target system (which are required to be accessed as part of the performance test campaign). As part of the UoW Performance Test Assessment, Atos will work with Atos customer to map out any technical requirements that are necessary based on the technical environment. 20

27 17. Trial service A trial service is available via the UoW for Proof of Concept (PoC). Price of the PoC is given in section 6 Pricing. 21

28 22