Software Testing & Quality Assurance

Transcription

1 Top Information Technology Outsourcing Vendors Software Testing & Quality Assurance

2 2007 Rank Software Testing & QA Company 1 Lionbridge 2 UST Global/QA Labs 3 Hewlett Packard 4 EPAM 5 Infosys 6 Trigent 7 LogiGear 8 AztechSoft/Disha 9 IBM Global Testing 10 Wipro 11 Patni 12 Sierra Atlantic 13 AppLabs 14 RTL/Recommened Test Labs 15 Software Quality Systems/SQS Group 16 Accenture 17 ZenQA 18 CalSoftLabs 19 Bleum 20 Symbio Group 3

3 Survey Overview In 2007, the Black Book Software Testing & QA Client Industry user survey investigated over 800 contracts held by 2600 of the top spending banking corporations and organizations globally. 18 Key Performance Indicators (KPIs) or criteria were employed, scored on each respective vendor by client type and ranked on a 0 10 scale per KPI. Key Findings 2007 Software Testing & QA Vendors Top Five Honors 1. Lionbridge 2. UST Global ( 3. Hewlett Packard 4. EPAM 5. Infosys Innovation and Deployment are the most important attributes influencing Software Testing & QA client satisfaction with their 2007 outsourcing providers. Strong dissatisfaction is uncommon in this niche ITO sector, occurring in only 6.7% of Healthcare Industry client types, 11.4% in Software & High Tech Company clients, and 4.5% of Education clients globally Single-vendors offering comprehensive QA services to services software testing clients ranked highest in the overall survey by clients company is maintained. During the audit, each data set is reviewed by a Brown-Wilson executive and at least two other people. In this way, our clients are able to clearly see how a vendor is truly performing. The 18 criteria on operational excellence are subdivided by the client's industry, market size, geography and function outsourced and reported accordingly. Black Book collects ballot results on 20 performance areas of consultation excellence to rank Advisors by outsourced practice outcomes and subjected to the same audit and validation processes. Situational studies are conducted on areas of high interest such as Green Outsourcers, Educational Providers in Outsourcing, Outsourcing Benchmarkers and Boutique Advisors. These specific survey areas range from four to twenty questions or criteria each. Statistical Confidence of Black Book Data Statistical confidence for each performance rating is based upon the number of organizations scoring the outsourcing service. Black Book identifies data confidence by one of several means: 1. Top Ten ranked vendors and advisors must have a minimum of five unique clients represented. Broad categories require a minimum of ten unique client ballots. 2. Vendors and Advisors with over ten unique client votes are eligible for top ten rankings and are assured to have highest confidence and lowest variation. Confidence increases as more organizations report on their outsourcing vendor. Data reported in this form is shown with a 95 percent confidence level (+/-.25,.20 or.15, respectively). 3. Raw numbers include the quantity of completed surveys and the number of unique organizations contributing the data for the survey pool of interest. Black Book Methodology How the Data Is Collected Black Book utilizes a three-step process to collect candid performance data. First, Black Book collects a series of direct evaluations covering 26 performance areas on leadership and senior management, which comprise the scored ranking of the Black Book 50 Best Managed Vendors. Second, Black Book collects ballot results on 18 performance areas of operational excellence to rank vendors by outsourced service lines. Third, the gathered data is subjected immediately to an internal and external audit to verify completeness and accuracy, and to make sure the respondent is valid while ensuring the anonymity of the client Who Participates in the Black Book Ranking Process? Over 120,000 individuals are invited to participate annually (including C officers of the Fortune 2000, Inc 500, institutional members and officers of various professional organizations, subscribers of our media partners and previously validated survey participants). Non-invitation receiving participants must complete a verifiable profile, utilize valid corporate address and are then included as well. Over 22,000 users were validated in the 2007 ranking process. The Black Book survey web instrument is open to respondents and new participants each year from March 1 to May 7 at 4

4 2007 Software Testing & QA Industry Outsourcing Vendor Rankings and Results 2688 qualified respondents ranked 148 software testing services suppliers offering some 845 contractual arrangements for testing and or quality assurance service processes as part of the Black Book annual survey conducted Q via web survey instrument. The Next Generation The focus on faster and cheaper in the outsourced software industry is no longer in vogue. Clients are looking for top line contributions, partnerships and extensive capabilities into testing expertise. black box testing methodologies. Test automation framework includes: Study of development process and solution architecture Automation test strategy Selection of appropriate testing tools Test automation implementation Customized test automation tools for specific client requirements. Security Testing Many enterprise applications and data are vulnerable to external and internal threats. Qualified vendors provide security-testing services for software products, applications and enterprise data. Security testing services include application vulnerability assessment and Risk identification. Software Risk Management & Analysis Customized software risk management services encompass project, product and process related risks. Software risk management framework includes: Development lifecycle study Risk identification Risk analysis Risk mitigation strategy & Implementation Risk monitoring Full Lifecycle Testing Software testing at every stage of software development life cycle can prevent defect occurrence and ensure reliable software development for businesses. Next generation software testing provides full life cycle testing services for product and application development life cycles. Black Box Testing Black box testing methodology covers the following: Functionality testing GUI testing Performance testing Stress and load testing Compatibility testing Installation testing White Box Testing White box testing improves the testability of a software product by making testing more effective and efficient. Top vendors provide white box testing services for applications developed using any language. Services include: Code coverage Path coverage Code analysis Test Automation Automation testing drastically reduces testing cycle times. Test automation services encompass white-box as well as 5

5 High Growth Sectors Industries/Sectors to Expand Sales Outsourcing Initiatives > 10% Client-Predicted Sectors of High Growth For Outsourced Software Testing 2008 Percent Increasing Initiatives in next 12 months Education 92.8% Healthcare 90.5% Computer Services & High Tech, Software 85.8% Pharmaceutical/Biotech/Life Sciences 84.6% Media & Entertainment 80.1% Consumer Packaged Goods 74.0% Manufacturing 69.8% Retail 58.4% Engineering, Architecture, Construction 53.2% Transportation & Supply Chain 44.9% Hospitality & Leisure 37.5% Banking, Financial & Capital Markets 25.5% Telecommunications 25.1% 6

6 Operational Excellence of Outsourced Software Testing & QA Firms Comprehensive Outsourced Software Testing & QA Vendor defined as being comprised of three surveyed functions: 1. Independent Analysis, Benchmarking, & Certification 2. Product Testing & Quality Assurance 3. Software Testing & Functional Performance Generally dissatisfied Does not meet expectations Cannot recommend vendor Inconsistently satisfied Meets some/few expectations Reserves recommendation Satisfactory performance Meets expectations Recommends vendor Overwhelming satisfaction Exceeds expectations Highly recommended vendor Color-coded Stop Light Dashboard Scoring Key Green (Top 10%) Scores better than 90% of Vendors. Green Coded Vendors received constantly highest client satisfaction scores. Clear (Top 33%) 6.94 to 8.00 Scores better than 67% of Vendors. Well scored vendors that have middle of the pack results. Yellow 5.00 to 6.93Scores better than 50%of vendors. Cautionary performance scores. Areas of improvement required. Red Less than 4.99 Scores worse than 66% of Vendors. Poor performances reported, Potential Cause for Contract Cancellation. 7

7 Vendors Grouped by Overall Score Mean & Corresponding Ranking TOP MEAN RANK PROVIDERS 1 Lionbridge UST Global/QA Labs Hewlett Packard EPAM Infosys Trigent LogiGear Aztechsoft/Disha IBM Global Testing Wipro Patni SierraAtlantic AppLabs RTL Testlabs SQS GROUP Accenture ZenQA CalSoftLabs Bleum Symbio Group 5.98 Outsourced Software Testing & QA Rankings TOP TIER VENDORS 1-6 UPPER MIDDLE TIER VENDORS 7-16 MIDDLE TIER VENDORS LOWER MIDDLE TIER VENDORS BOTTOM TIER VENDORS Ranked Firms Ranked Firms

8 Raw Scores and Mean by Criteria Questions Rank Outsourcing Testing Firm MEAN 1 Lionbridge UST Global/QA Labs Hewlett Packard EPAM Infosys Trigent LogiGear Aztechsoft/Disha IBM Global Testing Wipro Patni SierraAtlantic AppLabs RTL Testlabs SQS GROUP Accenture ZenQA CalSoftLabs Bleum Symbio Group

9 !! "!!#$%! &'$$$ ( #$ ) *$$+, - ( "!! %.*$$# /0 12 ( 3!!( (30 6 *#78, & 9#7$ 9 7$: -( 4 ;;+<7 1 5/+*+2+:'3<<:8 055==(((!! >5==((( (30,- (, - (( (! (,( ( - ( ( ( ( ( 10