TESTING SERVICES SERVICE DEFINITION G-CLOUD Commercial-in-Confidence civil.lockheedmartin.co.uk
SECTION 1 LOCKHEED MARTIN S TESTING SERVICES CAPABILITY OVERVIEW OF THE G-CLOUD SERVICE Lockheed Martin provides a full range of testing services to the public sector including the provision of: Full end to end Testing Services delivered either on site or off site; Test resource augmentation on site; Development of a Test Strategy for your organisation; Test Assurance (with a focus on knowledge transfer where appropriate); and Hosting of test environments and defect tracking systems. Lockheed Martin provides these services for a number of public sector clients including: Scottish Government; Skills Funding Agency; Ofqual; Department for Communities and Local Government; Transport Scotland; Marine Management Organisation; and Environment Agency. Our independent testing services follow ISEB (Information Systems Examination Board) guidelines and methods, and use a variety of best of breed test tools and methodologies. The Lockheed Martin testing team are trained in the use of a wide variety of testing tools. Test tools currently in use range from Open Source tools such as TestLink, JMeter, JUnit/NUnit, NCover/jCover to proprietary system tools such as automated test framework ODIN Axe, and web services testing tool SoapUI. Lockheed Martin utilises specialist tools and methods for accessibility and penetration testing, following industry best practices to validate against W3C and CESG. In addition to general testing capability, Lockheed Martin provides consultancy in all forms of testing and provides staff to undertake the following roles; Test Analyst, Test Team Leader\Coordinator and Manager. This encompasses the production of high quality test documentation, as well as bringing customer test teams up to the required experience and skill levels, ensuring effective knowledge transfer is achieved. All staff are versed in applying a suite of different test techniques to generate a rich and varied set of test scenarios. Boundary Value Analysis, State Transition Testing, Validation Testing, Syntactic testing and exploratory testing are examples of methods employed. Staff are experienced in working with requirements catalogues, product backlogs, interface documents, user stories, use cases, UML and ERD diagrams. Staff work closely with internal/supplier development teams and customer test teams, taking full part in agile scrum teams. All Test Analysts have excellent SQL querying and reporting capabilities. 2
TEST PHASES Testing covers all functional and non-functional aspects of the system deliverables. Functional testing starts early in the process with static reviews of systems documents and continues in parallel throughout the development process into formal testing cycles. Non-Functional aspects of the system such as performance, stress, soak, security and accessibility are risk-reviewed at all stages and tests developed to cover all such aspects and ensure that feedback is provided early in the life-cycle. Lockheed Martin has a mature risk based approach to testing accounting for the impact of failure and the probability of failure occurring, this influences the priorities for test execution and the choice of tests which are nominated for automation. Lockheed Marin provides test services to Public Sector customers across the following areas: Post-iteration System Testing (PST); Factory Acceptance Testing (FAT); Site Acceptance Testing (SAT); User Acceptance Testing (UAT); Operational Acceptance Testing (OAT); and Penetration Testing (PEN), this may be performed by an independent specialist 3rd party organisation. We offer these services in isolation or as a complete test cycle. PST This is a system integration test performed by the Lockheed Martin test team during a time-boxed iteration or sprint. Tests are executed against the new features contained in the release and automated systems tests are executed as a regression test of previous development work. Independent testing and customer access during the iterative development process ensures a fast feedback loop to the development teams. FAT This is a full system test performed by the Lockheed Martin test team on a test environment, against an agreed Acceptance Test Specification (ATS) covering both functional and non-functional requirements. The customer team is invited to witness this test activity. Typically the test environment has these characteristics: Lower technical specification; performance test results obtained on the environment must be caveated; and Elements of the full production environment cannot be replicated; simulated services and stubs may be deployed in order to evidence specific test steps. When FAT is complete, the Lockheed Martin Test Manager completes a Test Summary Report. SAT This is a full system test performed collaboratively by the Lockheed Martin test team and a customer team, at a customer site, against an agreed ATS. It constitutes one form of customer acceptance test, the other is UAT. When SAT is complete, the Test Manager completes a Test Summary Report and the customer is happy that the system can enter live usage (though the customer may also wish to perform UAT). 3
UAT This is a full system test performed by a customer team, at a customer site, where the customer tests the system in any manner they choose, within an agreed time-box. It constitutes one form of customer acceptance test (the other is SAT, see previous subsection). Whether or not the customer has their own ATS is irrelevant. Lockheed Martin supports customers in this activity, either by forming part of the test team or in the provision of consultancy services in guiding the customer team through the UAT process. OAT OAT is where the customer may choose to undertake full performance/non-functional testing on a production or pre-production system post or pre go-live. This can be an extension of UAT, carried out in parallel to UAT or a completely separate phase. Lockheed Martin can conduct this testing on behalf of customers using leading edge testing tools and techniques. TEST DELIVERABLES The following items are typical key Lockheed Martin deliverables provided as outputs from our engagement: Test Strategy/Approach defines the approach to testing for the project, the phases to be performed, exit criteria; Test Plan, one of these per test cycle, defines the resources to be allocated to a test cycle, the test areas to be covered and the schedule to be followed for testing; Acceptance Test Specification (ATS) including Test to Requirements traceability matrix defines the tests to be conducted to confirm both the functional and non-functional aspects of the system; Test Results Report A report detailing progress through testing, tests passed/failed/ blocked and the associated incidents raised; and Test Summary Report one of these per FAT phase, reports the results of testing, the number of incidents raised in each test cycle and the final position against the exit criteria. TEST ENVIRONMENT HOSTING Lockheed Martin provide hosting of test environments and defect tracking systems with remote access available to customers. We have our own UK data centre where we can build and deploy multiple n-tier server farm test environments as required. Within our projects we use JIRA and BugTracker to track both development and production issues. Both systems are available externally to customers to support the test and development process. 4
SECTION 2 CASE STUDIES ORGANISATION NAME CALEDONIAN MACBRAYNE (CALMAC) LIMITED GOODS / SERVICES / WORK SUPPLIED Calmac is committed to its part in providing and improving transport services, including the experience, to people in island and remote communities; supporting the aims of the Scottish Government and to making a good commercial return for its shareholders. In order to continue improving the efficiency of the service and the public s experience of travelling with CalMac Ferries, it is investing in a new Service Management tool supplied by a third party and referred to as Service-Now. This project is not solely aimed at delivering a technology based solution. In the wider context of this project, CalMac Ferries wants to transform the way it works with streamlined processes and stronger governance. The Service-Now project was separated into a number work packages; with this first work package being the implementation of the Incident Management module. Lockheed Martin was contracted to provide Calmac with a Test Manager who was tasked to fully the Incident Management module of Service-Now for its business within a three week period. As there were well over 500 permutations of the basic Type, Category and Sub- Category options for specific Incidents, Lockheed Martin s Test Strategy focused on testing the key areas and appropriate levels to determine if there are any major defects which require to be corrected prior to going Live. As the number of permutations for test were so high it was highly time consuming to undertake tests for all scenarios. It was essential therefore that the testing is approached in both a family orientated and agile manner. Our Test Manager successfully produced an Acceptance Test Specification which defined the agreed scenarios, including test steps and expected results. We also provided training and coaching to Calmac s business-resourced test team who conducted the testing directly. Ultimately Service-Now was suitably tested within the defined period and subsequently went live as per the initial project plan. Work is now underway to extend its use to include Service Requests and an Asset Inventory. 5
ORGANISATION NAME SCOTTISH HOUSING REGULATOR (SHR) GOODS / SERVICES / WORK SUPPLIED The SHR is working with its contracted main supplier and two sub-contractors to deliver a new Enterprise Content Management (ECM) system. The ECM system was delivered in a phased manner and provided the Regulator with a series of capabilities that underpins its core operation. These capabilities relate to data capture, data validation, business intelligence, document management, customer relationship management and workflow. The core software stack being deployed is Microsoft SharePoint, Microsoft Dynamics and Microsoft SQL Server. In addition a customisable forms generator component named CC Forms being deployed to specifically support data capture and data validation requirements. The SHR was established an agency light model. As such it did not have an in-house technology or test function. To enable the successful delivery of the ECM project there was a requirement to supplement the Regulator with addition capability (process, skills, resource) in the area of user acceptance testing. SHR therefore requested that Lockheed Martin provide a Test Manager resource to support this important phase of the project. Lockheed Martin s Test Manager worked directly alongside SHR s Project Manager to: Understand and agree the business scenarios delivered by the solution; Produce a Test Strategy and Test Plan aligned to the software deliveries which were being issued by SHR s supplier iteratively; Work with SHR s supplier(s) to document and agree the processes for fault identification, documentation and resolution; Produce an Acceptance Test Specification detailing the individual test scenarios to be tested by SHR with associated steps and expected results throughout; and Understand the SHR Test resource requirements bolstered by Lockheed Martin Testing resources to enable suitable levels of testing against a defined Project Plan. 6
ORGANISATION NAME SKILLS FUNDING AGENCY GOODS / SERVICES / WORK SUPPLIED Lockheed delivered the Data Collection and Funding Transformation (DCFT) solution for the Skills Funding Agency. Fundamentally, the solution was designed to be scalable, performant and resilient from the outset. As part of the overall delivery contract, Lockheed Martin had a requirement to conduct formal performance testing. There are 9 main types of performance testing that Lockheed Martin considered, and a subset of these was agreed with the Skills Funding Agency for DCFT, with agreed test criteria: 1. Single User To measure metrics when only a single user is exercising the system; 2. Load To measure metrics when the system in under an acceptable operational load. In the case of the DCFT system, it is expected to sustain a maximum load of concurrent users (as detailed in the Non-Functional Requirements Catalogue); 3. Soak A load test that is executed for an extended period of time. The purpose of this test is to verify the stability of the performance metrics; 4. Stress To measure metrics when a system has exceeded a specified operating limit; 5. Resilience To measure metrics during transitions from stress to load conditions; 6. Failure To measure metrics at the point where a system is no longer able to functionally operate correctly; 7. Concurrency To measure the maximum simultaneous threads of execution (simultaneous user requests) that a system can support. This type of performance testing can be particularly useful for message passing servers; 8. Throughput To measure the transactional rate of the system and identify the point of maximum throughput without regard to individual transaction times; 9. Recovery To measure metrics during the transition from failure back to stress conditions. A number of different tools were used for performance and load testing, with the tools varying depending on the individual circumstances being tested. Lockheed Martin typically used Visual Studio Test Editions for Microsoft solutions, along with SoapUI, loadui and JMeter. 7
SECTION 3 PRICING FOR SPECIALIST SERVICES PRICING STRUCTURE This section shows details of the pricing structure for all aspects of Specialist Services. Level 1 SFIA Alignment Area of Capability 1. Follow 2. Assist 3. Apply 4. Enable 5. Ensure/Advise 6. Initiate/Influence 7. Set Strategy/Inspire STRATEGY AND ARCHITECTURE BUSINESS CHANGE BUSINESS CHANGE IT Consultancy 570 665 750 793 900 Project/Programme Management Business Analysis 435 470 570 665 750 665 750 793 900 SOLUTION DEVELOPMENT AND IMPLEMENTATION Application Development 435 470 570 665 750 793 900 Testing 435 470 570 665 750 SERVICE MANAGEMENT Service Management 435 470 570 665 750 793 900 Application Management 435 470 570 665 750 Transition Management 435 470 570 665 750 793 900 The definitions of the elements in the table indicated as 1 7 are shown below for reference. 8
FOLLOW Works under close supervision. Uses little discretion. Is expected to seek guidance in expected situations. ASSIST Works under routine supervision. Uses minor discretion in resolving problems or enquiries. Works without frequent reference to others. APPLY Works under general supervision. Uses discretion in identifying and resolving complex problems and assignments. Usually receives specific instructions and has work reviewed at frequent milestones. Determines when issues should be escalated to a higher level ENABLE Works under general direction within a clear framework of accountability. Exercises substantial personal responsibility and autonomy. Plans own work to meet given objectives and processes. ENSURE/ADVISE Works under broad direction. Is fully accountable for own technical work and/or project/ supervisory responsibilities. Receives assignments in the form of objectives. Establishes own milestones and team objectives, and delegates responsibilities. Work is typically self-initiated. INITIATE/ INFLUENCE Has defined authority and responsibility for a significant area of work, including technical, financial and quality aspects. Establishes organisational objectives and delegates responsibilities. Is accountable for actions and decisions taken by self and subordinates. SET STRATEGY/ INSPIRE Has authority and responsibility for all aspects of a significant area of work, including policy formation and application. Is fully accountable for actions taken and decisions made, both by self and subordinates. STANDARDS FOR DAY RATE CARDS The rates do not include VAT, which would be added at the rate prevailing at the time. The rates do not include expenses, which would be charged if required, at cost, in addition, and in accordance with customer staff expenses rates. The rates assume a 7.5 hour working day (although all of our consultants are flexible to some degree, to ensure peaks of work are met). The working week is from Monday to Friday excluding national holidays. OFFERED DISCOUNTS Lockheed Martin is willing to offer the following wide range of discounts. Prompt Payment: We offer a discount on the payment within 15 days of a fully authorised invoice. The discount is typically in the region of 0.5%, and we offer a sliding scale of discount until the 30 day mark after which there is no discount available for prompt payment; Payment in Advance: If a customer pays some or all of the assignment price at the outset of the assignment then we offer a discount on the percentage of the final fee paid in advance. For example, for payment of 100% of the fee in advance we discount 1% down a sliding scale of 0.1% for 10% paid in advance; 9
Volume Number of days: If a particular assignment or group of assignments for a customer falls into a set band of days then a discount is applied to the total cost. The days are grouped in 100-day bands with 500+ being the upper band. Discounts are presented on a sliding scale from 1.5% to 5% (See table below); Volume Expenditure: An alternative to the number of days is the level of expenditure. This works in exactly the same way as the Number of Days discount except the bands are monetary values. The values are grouped into 75,000 bands with the upper band in this case is 375,000+. The range of discount is the same as the Volume - Number of Days discounts (See table below); Time length of assignment: If at the outset of an assignment the customer commits to a set number of days then we discount on a basis similar to that for Volume - Number of Days, the difference here is that the commitment is made at the beginning of the project and the discount is greater for the customer; Return of unused money/resources: If a customer has paid a fixed price for an estimated number of days work, and it becomes clear that the work will take significantly less than estimated then we only invoice for an agreed number of days; and Follow-on work: For work which is an immediate follow-on of a previous contract with no break in the assignment we offer a discount of 1.5%. EFFORT/FIXED PRICE COST DISCOUNT OVER AGREED RATES COMMENT <100 days / 74,999 100-199 days / 75,000-149,999 200-299 days / 150,000-224,999 300 399 days / 225,000-299,999 400 499 days / 300,000 to 374,999 >500 days / > 375,000 0% Can be considered on an individual basis 1.5% Discount not applied retrospectively 2.25% Discount not applied retrospectively 3.5% Discount not applied retrospectively 4.25% Discount not applied retrospectively 5% 10
SECTION 4 GENERAL ENGAGING WITH LOCKHEED MARTIN At Lockheed Martin it is our mission to make engagement with us as simple as possible. You can contact Lockheed Martin through one of the following methods: Email: gcloud@civil.lmco.com Telephone: 0141 814 3700 (ask for Morag Young our GCloud Services Manager or Craig Donnelly, our Public Sector Sales Director) You can follow us on Twitter @lmuktweets We are always keen to engage with you to discuss your requirements even if these are at a very early stage. Lockheed Martin offers a free of charge discovery meeting to discuss your requirements and to set out how we are able to assist your Organisation. This applies across the full range of services offered in our Catalogue on Cloudstore. We honour the following Service Level commitments in relation to any work we are requested to undertake: We will schedule a discovery session with you within 5 days of contacting us; We respond to all requests for quotations within two working days of receipt of the enquiry; Once we agree the requirements with yourselves, Lockheed Martin produces and returns a detailed proposal within five working days; and Depending upon the size and scale of the assignment, Lockheed Martin endeavours to commence work within ten working days of formal proposal acceptance. Our engagement model is flexible to suit many different scenarios, we are happy to provide our services on the following (or any combination thereof): Fixed price; Time & Materials; Target Price; On-site individuals or teams; and Blended delivery (customer and Lockheed Martin teams). Lockheed Martin has no minimum length of engagement or order value for the work that we undertake. ORDERING AND INVOICING Lockheed Martin accepts electronic orders and provides electronic invoices. Please contact us at the email address above if you would like to discuss this further.. For any orders placed, Lockheed Martin provides order confirmation which includes a detailed cost breakdown. Prior to the commencement of any work secured under the G-Cloud framework, Lockheed Martin request that you provide a customer acceptance of the order and also the completion of a Call-Off Contract. We are happy to commence work on a Letter of Intent which indicates that orders and call-offs are being prepared. Lockheed Martin will agree the invoicing frequency and mechanism in advance of any assignment commencing. 11
TERMS AND CONDITIONS For a full example of Lockheed Martin s terms and conditions please refer to the Lockheed Martin Terms & Conditions for Cloud Services document. Depending on the type of service being procured, additional elements such as licences and support may need to be incorporated into the terms and conditions. We are also happy to use your terms and conditions as a basis for a contract. CUSTOMER RESPONSIBILITIES For any specified call off requirement, the dependencies on the customer associated with the work (which could differ in each case) are discussed and agreed prior to the commencement of the work, and form part of the Call Off contract. 12
WE RE A BIT DIFFERENT AT LOCKHEED MARTIN We re driven to be better at everything we do. Our mission is simple - to be the best provider, employer and investment for our customers, employees and investors alike. To achieve this, we need to continually deliver on our vision of being the leading global provider of business technology solutions to the Energy, Transport and Public Service sectors. Alongside our ambitious targets, it s our core values that are the guiding principles that govern the way we do business and provide a blueprint for a more rewarding way to work for our employees. Our core values should be evident in everything we do. CONTACT US Call us on +44 (0)141 814 3700 Email us at: gcloud@civil.lmco.com GET SOCIAL twitter.com/lmuktweets linkedin.com/company/lockheed-martin civil.lockheedmartin.co.uk