Automated Software Quality Vendor Profile: With Mobile and Load Testing, Keynote Evolves Company

Transcription

1 VENDOR PROFILE Automated Software Quality Vendor Profile: With Mobile and Load Testing, Keynote Evolves Company Melinda-Carol Ballou IDC OPINION Global Headquarters: 5 Speen Street Framingham, MA USA P F Software drives business innovation and performance optimization. Performance optimization and mobile branding have become a key success factor for competitive position. At the same time, organizations are leveraging automated software quality (ASQ) software-as-a-service (SaaS) offerings and the cloud for testing purposes increasingly. The ability to access load testing as a service and also adaptive, flexible infrastructure to support testing environments to respond to dynamically changing business needs and buying cycles is a core benefit. And the emergence of a variety of software deployment environments demanding fast cycle times from mobile to embedded to traditional Web and client server drives the need for additional support for test infrastructure and device testing. IDC sees these key trends driving growth for cloud testing: In a volatile economy as companies reinvest, financial constraints, global competition, and innovation are driving comfort levels with software as a service and cloud testing. These enable rapid access to both quality solutions and support infrastructure to sustain complex software sourcing and distributed development. SaaS and cloud testing require fewer resources and less capital expenditure compared with on-premise and facilitate better quality and faster turnaround times. These combined factors and others that push both complexity and the need for speed (without loss of quality) are driving users to adopt automation solutions for cloud and for mobile testing. Small and medium-sized businesses (SMBs) and managed service providers are leveraging cloud testing to lower costs for testing where the investment demand for on-premise solutions is too exorbitant for these smaller companies and targeting mobile platform testing. Large service and solution providers see ASQ SaaS and cloud testing as an opportunity to evolve and bundle new, quality services together to target emerging markets for additional value-added and service offerings. This complements well the quality needs for mobile, ERP, security, and other areas (regulatory compliance, VoIP, etc.). Benefits of self-service provisioning impact cost and increase adaptability, flexibility, and dynamism. With the advent of agile approaches to both development and product evolution, including apps for mobile environments, software release cycles and churn are faster than ever. Business-critical branding and demand for mobile applications necessitate speedy turnaround. Compressed release times from days to hours drive the need for infrastructure availability and automation. Filing Information: November 2012, IDC #237730, Volume: 1 Application Life-Cycle Management: Vendor Profile

2 IN THIS VENDOR PROFILE This IDC Vendor Profile is one in a series of IDC studies that examine automated software quality (ASQ) solutions and provides both vendor analysis and customer reference context for technology adoption. This Vendor Profile assesses the ASQ strategy and position of Keynote, a vendor targeting the application performance management, load testing, and mobile testing arenas. Here, we discuss Keynote's LoadPro load testing software-as-a-service (SaaS) offering and the company's DeviceAnywhere mobile device testing solutions. We position Keynote within this product category and relative to trends in user adoption for the dynamic mobile and load testing market and in the context of an overall economy that remains volatile and demands efficient access to software quality and deployment. We also include summaries of three Keynote customer references: a banking organization and a computer systems company, both of which made use of Keynote's mobile testing solutions with Keynote DeviceAnywhere, and an eretailer, which used Keynote's LoadPro load testing services. SITUATION OVERVIEW Company Overview Products: Keynote DeviceAnywhere Test Center Enterprise, Keynote LoadPro, and Keynote Test Perspective Keynote is headquartered in San Mateo, California, with around 450 employees. In just the ASQ market, the company has a focus on mobile and Web SaaS testing in the cloud (also with on-premise and hybrid offerings). Keynote experienced combined ASQ revenue of $26.5 million for CY11 (up 14.5% from $23.1 million in 2010), including backstreamed revenue from Keynote's DeviceAnywhere acquisition (4Q11) and preexisting load testing capabilities, per IDC estimates this is out of the total combined revenue for CY11 of $111 million. With a customer base of around 1,000 for DeviceAnywhere, 2,000 users for the company's Keynote Perspective products (including application performance management for Web and mobile platforms as well as load testing), and 4,000 overall including its Keynote SIGOS services (which provides testing of mobile networks, services, and components), Keynote has a good established base from which to grow its combined business as the company targets a growing mobile and Web market on the cloud. Keynote invested early and strongly in the mobile space with its DeviceAnywhere acquisition in 2011 and is effectively executing in that arena. As corporate branding and presence are increasingly driven by mobile platforms and applications, quality needs for device testing and mobile applications are becoming business critical, and savvy customer awareness presaged solution availability. As broader user experience underpins emerging and increased demand and adoption for mobile testing, IDC expects significant growth in this area throughout the upcoming forecast period ( ). In that context, Keynote's investments in mobile device testing and load testing (as part of the company's application performance management suite) position the company well moving forward. (Keynote's most recent financial reports from CY 1H12 indicate growth for ASQ and emerging adoption for mobile.) Keynote is considered a Major Player in the IDC MarketScape for enterprise ASQ. 2 # IDC

3 Company Strategy Keynote's ASQ products include Keynote DeviceAnywhere Test Center Enterprise for mobile device testing as a primary ASQ focus, with additional testing product and turnkey service for load testing with Keynote LoadPro (as part of Keynote Perspective's core application performance management portfolio). Keynote also partners and integrates with additional ASQ and ALM providers, including HP for Quality Center and QuickTest Professional integration, IBM Rational, and SOASTA for cloud performance testing. (Keynote has extensive partnerships with SIs that have evolved mobile practices leveraging Keynote capabilities.) Keynote LoadPro enables performance load testing in conjunction with Keynote's application performance management solutions. Keynote's DeviceAnywhere Test Center Enterprise SaaS solution has a centralized server and database, hosting user information and test assets in the cloud. Users access the system through a client application called Keynote DeviceAnywhere Studio, which is a Java Swing application. Mobile devices are connected to and deployed into the cloud and are accessible to users in DeviceAnywhere Studio, via an Internet connection. Dedicated systems can also be supported, with a separate hardware and software stack (private database, central server, devices) for the given customer. Differentiators for Keynote's ASQ product suite include SaaS/cloud or on-premise support, access to a vast array of mobile devices over the Internet, and support for creating and scheduling automated tests on real mobile devices. Keynote's suite of products facilitate developing and launching mobile applications using device test strategy capabilities, with manual and automated test solutions along with postrelease monitoring and analysis tools and test planning modules. Keynote offers mobile platform, device planning, and testing planning via manual and automated testing and monitoring on the back end and is in the process of enhancing analytics capabilities that come with monitoring. In addition, Keynote offers "Just in Time" delivery and flexible support for dynamic needs for mobile devices. Although the primary focus for IDC MarketScape is not mobile testing, we evaluate this area as part of our enterprise ASQ focus and will be releasing additional, targeted mobile research for market evaluation purposes. Keynote's Direct-to-Device technology allows customers to operate and run tests on real mobile devices over the Internet. This enables remote and distributed teams to access a test bed of devices. It also enables the QA team to spend more time testing and less time trying to track down physical devices and managing plans. This system supports testing on any device, including smartphones and tablets. The test automation solution allows customers to create hierarchical tests (test cycles made of test cases made of individual device actions) that enable one test cycle to run across many devices. Additional time-saving features include a scheduler so that users can schedule tests to run overnight, with all results waiting for them when they return in the morning, and an integration with build systems so that a suite of tests can automatically be triggered with a new build. In addition, DeviceAnywhere integrates with established test management tools such as HP Quality Center and IBM Rational Quality Manager so that customers that are familiar with and have adopted these tools can continue to use them to manage their test process IDC #

4 Keynote Test Perspective is a cost-effective, self-service Web site load testing tool that lets users run realistic load tests on demand to receive immediate feedback about performance impact resulting from Web site changes or modifications. Sample scripts and processes facilitate setup, unified dashboard enables individual and team visibility, and Keynote provides consistent and repeatable load conditions via its Global Testing and Monitoring Network. Keynote LoadPro leverages this technology and its consulting team expertise to provide a turnkey service for Web site load testing. FUTURE OUTLOOK Opportunities exist for Keynote to more closely integrate across the company's portfolio of products, which it is in the process of evolving and also for functional testing of mobile applications. Strong synergies exist for optimization of mobile applications and leverage of the application monitoring capabilities of Keynote's Perspective products. Indeed, IDC saw initial coordination and interest as part of the reference discussion with Keynote customers across areas. User Reference Summary IDC contacted Keynote references for both mobile testing with Keynote DeviceAnywhere and load testing with Keynote LoadPro services. With the pressing need to provide mobile access to customers, the DeviceAnywhere customers leveraged the products to drive device testing and access across multiple devices to speed delivery of core mobile applications and to cut costs. One banking organization had no option but to create mobile access to its banking and financial services business. As the organization evaluated options, DeviceAnywhere was one of the few solution providers available to provide key mobile testing capabilities. A growing need at the bank to test mobile applications across a broad range of devices to discover problems drove the bank's decision to invest in DeviceAnywhere. This was a "must-have" purchase, given competitive pressures at this financial institution to provide support for its services to customers across an array of mobile platforms. Similarly, a major computer systems provider with 100,000 employees and 50 different parts of the organization investing in mobile applications and focus also purchased DeviceAnywhere to provide consistent support for its mobile strategy. Both companies were able to effectively automate their mobile device testing initiatives, cutting time to market dramatically and enabling management, prioritization, and targeted focus to what had been dispersed, expensive, and inefficient mobile development and release strategies. Immediate access to appropriate devices enabled the banking organization to increase efficiencies via access to appropriate devices to reproduce bugs, to prioritize planning, and to retest and redeploy. This is particularly important, given the range of mobile devices they must support. (BYOD and competitive pressures are driving demand for mobile breadth, and the time and resource costs to provide broad device support internally are prohibitive.) Keynote DeviceAnywhere also enables the computer systems provider organization to set up additional groups quickly for mobile testing and device access. A challenge for this organization is the learning curve for adopting new mobile devices and 4 # IDC

5 seeking more efficient ways to limit how much content it tests on the mobile applications. However, the ease of access and the ability to build in mobile testing as part of a broader life cycle of mobile application from inception to deployment have been helpful as well as moving to a more consistent process for mobile testing. (These companies suggest including at least two rounds of device testing, with the second round devoted to bug testing, in coordination with mobile device providers.) A major eretail organization invested in Keynote's LoadPro to enable performance testing services for its key retail applications on which the company's revenue depends. A challenge for this organization was that the company started off primarily as a retailer but had to become more and more of a software company, despite dramatic underfunding for up-front investment in IT infrastructure and poor initial software creation with quality problems. (For instance, load testing was an afterthought.) Code bases would crash frequently, and during preholiday season 2010, the company sought to dramatically improve performance to address huge spikes in demand in the year-end (which drove 40 50% of the company's business). This company opted for Keynote because of Keynote's strong testing services and partnership-oriented model and reliable product portfolio to get the company through its code quality and performance problems. (The company chose Keynote for load in conjunction with Raxx base for Web polling across locations.) As part of the decisionmaking process, the company said that Keynote was more service and partner oriented and less sales oriented than competitive products in which the eretail organization had unsuccessfully invested earlier and that Keynote also had significantly lower price points for comparable products. Results for this organization included getting through the 2010 holiday season successfully with no downtime via improvements and reworking of the existing software and leveraging Keynote to test a completely new system in 2011 for successful execution during that holiday period. In 2010, the company triaged the old system and then completely rebuilt for 2011 testing and was able to mitigate risk and minimize exposure successfully. The company purchased service support from Keynote and ended up testing 80,000 sessions per hour with roughly 20 clicks (ratcheted up from 25,000 earlier) in realtime production server environments to get through the 2010 holiday season. (The prior year in 2009, the site was down for 36 hours at a cost of $50,000 per hour.) In 2010, the initial tests of the company's code didn't go smoothly, the software failed the first test, and the company had to make changes including architectural changes that it hadn't anticipated (in memory issues, parallel processors were resource hogs, using too many CPUs, synchronous thread issues for Web, etc.). The company logged all results and was able to do a capture of the thread pool as it was being hammered; the company could go over line by line from its hang dump to create a memory profile of how the system was working. It became apparent to the company that for the parallel aspect of the system, parts of it needed to be reworked completely, which would have been impossible without load testing with Keynote. Improvements sought with Keynote by this reference included a more intuitive Web portal, with pointers to problematic code areas and better, deeper report automation. This company views Keynote as a preferred partner for load, and it was able to do performance testing in conjunction with stable release builds and then do a freeze for significant features for its B2C applications to achieve business-critical goals for software performance during the holiday season IDC #

6 Across products, references valued Keynote for its product portfolio capabilities, support for key emerging mobile and existing Web environments, and proactive responsiveness to customer needs. ESSENTIAL GUIDANCE Advice for Users Users should engage in gap analysis assessments to determine core needs for ASQ across core emerging areas (such as mobile, ASQ SaaS, cloud testing) and evaluate and adopt appropriate automated technology and testing services. Technology alone is insufficient; however, effective adoption of automated solutions must be accompanied by appropriate organizational and process change to help ensure consistent usage. This is even more the case in core emerging business-critical areas such as mobile branding and application deployment. As customer examples indicated, these are "must-have" deployment environments to reach and sustain customer and partner relationships. We expect mobile device testing to be accompanied with effective performance optimization and application performance management support, as well as effective application testing. LEARN MORE Related Research IDC MarketScape: Worldwide Cloud Testing and ASQ SaaS Vendor Analysis Enabling Business Agility and Quality in the Cloud (IDC #237597, forthcoming) IDC MarketScape: Worldwide Enterprise Automated Software Quality Vendor Analysis Driving Multimodal Innovation with Testing Portfolio Strategies (IDC #236616, October 2012) Worldwide Cloud Testing and ASQ Software as a Service Forecast and 2011 Vendor Shares: Enabling Multimodal Quality and Business Agility with Cloud Testing (IDC #236928, September 2012) Worldwide Software Quality Analysis and Measurement Forecast and 2011 Vendor Shares: Code Analysis Enables Multimodal Businesses and Security (IDC #236836, September 2012) Worldwide Automated Software Quality 2011 Vendor Shares: Innovation Demands Range of Quality Responses Driving Growth (IDC #235455, June 2012) Worldwide Software Configuration Management Forecast and 2011 Vendor Shares: Excellent Combined Growth as Multimodal Complexity Demands Governance (IDC #235479, June 2012) 6 # IDC

7 Worldwide Automated Software Quality Forecast: Multimodal Development with Mobile, Social, and Cloud Drives ASQ Growth (IDC #235457, June 2012) IDC's Software Taxonomy, 2012 (IDC #235401, June 2012) Copyright Notice This IDC research document was published as part of an IDC continuous intelligence service, providing written research, analyst interactions, telebriefings, and conferences. Visit to learn more about IDC subscription and consulting services. To view a list of IDC offices worldwide, visit Please contact the IDC Hotline at , ext (or ) or sales@idc.com for information on applying the price of this document toward the purchase of an IDC service or for information on additional copies or Web rights. Copyright 2012 IDC. Reproduction is forbidden unless authorized. All rights reserved IDC #