FOR: Application Development & Delivery professionals The Forrester Wave : Application Life- Cycle Management, Q4 2012 by Tom Grant, ph.d., October 23, 2012 Updated: October 24, 2012 Key TaKeaWays Leaders Build product strategy around alm Use Cases Forrester Wave evaluation Leaders IBM, Rally Software, PTC, CollabNet, Microsoft, and Serena Software offer strong solutions for ALM problems that no single point solution could address. Their product strategies focus on such challenges as delivery and the disruptions of Agile adoption, not just throwing new capabilities at the customer. strong performers and Contenders Find a specific audience Atlassian and Rocket Aldon don t focus as much on specific use cases that span many ALM activities. Instead, they build tools to address smaller but widespread problems, such as release management and developer collaboration. Meanwhile, HP is expanding its ALM offering beyond Quality Center. alm is Going Through a period of Redefinition and innovation Hot ALM issues such as DevOps, embedded software, and requirements definition show how the concept of ALM has expanded and changed. During this transition, the seemingly crowded ALM market will benefit from the innovation that comes from a wealth of competing and complementary product offerings. Forrester Research, Inc., 60 Acorn park Drive, cambridge, ma 02140 UsA Tel: +1 617.613.6000 Fax: +1 617.613.5000 www.forrester.com
October 23, 2012 Updated: October 24, 2012 Why Read This Report The Forrester Wave : Application Life-Cycle Management, Q4 2012 IBM, Rally Software, PTC, CollabNet, Microsoft, And Serena Software Lead The Pack, With HP, Atlassian, And Rocket Aldon Following by Tom Grant, Ph.D. with Kyle McNabb and Alissa Anderson In Forrester s 116-criteria evaluation of application life-cycle management (ALM) vendors, we identified the nine most significant software providers in the category Atlassian, CollabNet, HP, IBM, Microsoft, PTC, Rally Software, Rocket Aldon, and Serena Software and researched, analyzed, and scored them. This report details our findings about how well each vendor fulfills our criteria and where they stand in relation to each other to help application development and delivery professionals select the right partner for their software-fueled business innovation efforts. Table Of Contents 2 Innovation Drives Interest In Application Life- Cycle Management 6 10 14 17 Application Life-Cycle Management Evaluation Overview The ALM Forrester Wave Finds Many Leaders And Different Directions Vendor Profiles Supplemental Material Notes & Resources Forrester conducted lab-based evaluations in September and October 2011 and interviewed nine vendor and user companies: Atlassian, CollabNet, HP, IBM, Microsoft, PTC, Rally Software, Rocket Aldon, and Serena Software. Related Research Documents The Time Is Right For ALM 2.0+ October 19, 2010 The Forrester Wave : Agile Development Management Tools, Q2 2010 May 5, 2010 2012, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. To purchase reprints of this document, please email clientsupport@forrester.com. For additional information, go to www.forrester.com.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 2 INNOVATION DRIVEs INTEREST IN APPLICATION LIFE-CYCLE MANAGEMENT Business innovation now drives the application life-cycle management (ALM) market. The contribution that application development and delivery makes to a company s business goals making workers more productive, creating engaging customer experiences, and bringing new software products to market more successfully must be more direct and successful. Software increasingly plays a central role in a firm s ability to deliver new products and services or exploit new channels. Firms can no longer accept historical gulfs between business and application development and delivery teams as, increasingly, firms now expect to manage application development and delivery as a business and treat it as a competency. 1 Unfortunately, everyone has their long list of reasons why they struggle with better managing application development and delivery. The work is creative, and teams must figure out how to work with one another. Tools or techniques that impose an assembly-line model will fail. Application development and delivery still need management, but the tools of management must fit the nature of the creative and collaborative work. Software and application development work itself is changing. This Forrester Wave evaluation examines the most significant developments that affect both ALM strategy and ALM tools. For example, the definition of ALM has stretched to include not just development but also delivery (see Figure 1). Some of the vendors we assess have expanded their tools portfolio to add or enhance their support for release management and other delivery-related activities. The need to support the business management of application development and delivery underlies this Forrester Wave s evaluation criteria: Delivering business value, not just driving process efficiency. ALM s original aim was process improvement. Firms turned to ALM to do a better job of managing project timelines, dependencies, and deliverables; ensure that every requirement could be traced to a real software capability; and test to ensure that the software worked. While process improvement is still important, increasing efficiency in building and delivering software is as important a goal for ALM as proving the value of the software, as it contributes to business outcomes. Supporting individual, team, and institutional management. ALM s first wave of tools focused on in truth application development management (ADM), designed to increase developer and tester productivity. Now, firms seek ALM tools to improve team success while linking these teams to the larger application development and delivery activity within the organization or, in many cases (such as offshore partners), outside of it. Extending life cycles to include delivery. Organizations now recognize that the life cycle of ALM does not stop with check-in or build phases. The real life cycle and process goes on much longer, arguably all the way to the moment the application goes live and even beyond to support continuous feedback.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 3 Managing software that gets deployed everywhere. Software is truly everywhere, from data center servers to laptops, from mobile devices to cars and refrigerators. Regardless of where the software winds up, software development and delivery management is still necessary. Integrating across tools. Vendors have reluctantly admitted that customers are not interested in their suite of tools, no matter how impressive or cost effective they might be. Switching costs can be high, and many capabilities, such as issue tracking and source control, are so commoditized that switching makes little sense. Some tools, such as HP Quality Center, are so widely adopted that other vendors must integrate with them in some fashion. The speed of tool innovation in some areas, such as continuous delivery and requirements, outpaces what any single vendor can keep up with, providing another reason for integrating across tools. Using reporting and dashboards as instruments of change. Many application development and delivery organizations aspire to be learning organizations and see their investment in ALM tools in that light. The data in an ALM system provides a yardstick for a team s continuous improvement efforts. Some teams use this information to communicate with outside parties about the shape and value of new Agile and Lean practices. Larger organizations may use the same information to track software value streams across projects and teams. In these and other scenarios, reporting and dashboards provide the ultimate payoff for ALM investments. Figure 1 ALM Boundaries Expand To Encompass The Software Value Chain Portfolio management Project management Just-in-time demand management Production planning closed loop Build and software configuration management Release management Deployment Production control closed loop Change-aware continuous integration Testing and quality assurance Change management Service management 60080 Source: Forrester Research, Inc.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 4 ALM Vendors Have Taken Many Steps Forward, But A Long Road Remains ALM tools have made significant progress since our last evaluation: ALM vendors have discovered macro differentiators. In past years, ALM vendors did far less in their product strategy and product marketing to clearly differentiate their products. With notable exceptions such as Rally Software, a company that built its ALM strategy around Agile support, ALM vendors had noticeable differences only if you carefully examined the details of their product features. Today, ALM vendors more clearly identify the types of customers they serve and the problems they can help these customers address. Serena Software, for example, tells a very clear story about the challenges orchestrating development and delivery in large IT organizations and how its products help organizations address those challenges. Vendors help improve the efficiency of the larger ALM process. As noted earlier, the boundaries of ALM have expanded to include more of the delivery phase. In response, ALM vendors have expanded their capabilities in this area to remove the inefficiencies that arbitrarily lengthen and complicate the delivery phase. Requirements, a part of ALM, need innovation attention. Recent innovations in requirements tools, such as visualization and social media integration, have increased the depth and accuracy of insights and the speed of collecting them. These innovations have occurred in requirements-focused companies such as Balsamiq Studios, Blueprint Software Systems, irise, and Visure Solutions. Even though ALM vendors provide requirements tools, they have not kept pace with these innovations. Mobile and embedded development scenarios get insufficient attention. Embedded software developers still get scant attention, with the exception of MKS (PTC) and IBM. Surprisingly, given how hot the mobile development market is, ALM vendors by and large do nothing special for mobile developers, including in areas such as testing where there is a large need for better tools. Collaboration is the lightning that ALM vendors can t capture in a bottle. Many of the biggest ALM challenges, such as coordinating the work across geographically distributed teams, boil down to collaboration problems. Collaboration presents some extremely vexing puzzles for tools vendors, particularly given how idiosyncratically a specific set of people may collaborate with each other. Still, some basic collaboration features desktop sharing for team members in different locations and wikis for capturing and updating information, for example are far less common than expected.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 5 The Market Is Still Open For Innovation And New Contenders The vendors we assessed in this evaluation provide an impressive array of tools with a broad range of capabilities. Nevertheless, they do not represent every possible innovation in the expanding and evolving realm of ALM. The ALM market is still wide open for anyone to devise a better way to address a problem, identify new and unmet challenges, or give a particular class of customers more attention than the bigger, general-purpose vendors can. Vendors listed in this section were not evaluated but represent various innovative approaches to ALM, demonstrating how much room remains within this market for innovation and specialization: Blueprint focuses on requirements. Some aspects of ALM pose tough challenges that don t have an easy solution. Smaller vendors focused exclusively on some of these challenges can innovate in ways that larger suite vendors may not. Blueprint, a requirements tools vendor, provides a good example of the important role that these smaller companies with a much narrower focus play in today s ALM market. Hansoft caters to video game developers. Since the video game market is nearly as large as the movie industry, it should be no surprise that at least one vendor, Hansoft, has identified this market as its core constituency. Polarion tackles difficult compliance challenges. Few of the vendors we evaluated have taken steps to address complex compliance requirements, leaving no small amount of opportunity for a smaller vendor such as Polarion Software. SmartBear Software makes quality the starting point. While many ALM vendors are shifting their attention to later parts of the development and delivery process, SmartBear Software makes those stages the starting point for its product strategy. This strategy includes both code quality improvement through review and testing and quality of service maintained through web monitoring. Tasktop is the Switzerland of integration. Each ALM vendor has its own set of integrations with other tools, both open source and commercial. Each ALM vendor also has its own approach to integration. Unsnarling this tangle sometimes requires going to a neutral third party, such as Tasktop, for cross-alm integration. ThoughtWorks provides thought leadership in continuous delivery. A real thought leader in continuous delivery, ThoughtWorks provides a modest range of tools but a very established consulting business (primarily aimed at helping with Agile adoption). VersionOne supports Agile. Rally has built Agile and Lean support into its tools, and other vendors such as Microsoft and IBM provide Agile-specific configurations. With Agile
The Forrester Wave : Application Life-Cycle Management, Q4 2012 6 continuing to spread, there is still room for another Agile-focused ALM vendor such as VersionOne to meet the needs of Agile teams in a different way than its competitors. APPLICATION LIFE-CYCLE MANAGEMENT EVALUATION OVERVIEW To assess the state of the application life-cycle management (ALM) market and see how the vendors stack up against each other, Forrester evaluated the strengths and weaknesses of nine top vendors. These companies fell into two broad categories: Large software companies that, among other products, also offer ALM tools. In this category, HP, IBM, and Microsoft maintain a strong position in the ALM market. Smaller software companies that specialize in ALM. Other companies included in this evaluation, such as Atlassian, CollabNet, Rally Software, Rocket Aldon, and Serena Software, have made ALM their exclusive business. Until its recent acquisition by PTC, MKS also fell into this group. Evaluation Criteria Measured Breadth Of Portfolio And Support For Use Cases After examining past research, user need assessments, and vendor and expert interviews, we developed a comprehensive set of evaluation criteria. We evaluated vendors against 116 criteria, which we grouped into three high-level buckets: Current offering. ALM covers a vast expanse of functionality for developers, business analysts, testers, project managers, and people in a wide array of other roles. Therefore, we developed more than 86 distinct criteria to measure the strength of each vendor s tools. Strategy. The value that a vendor brings to the table goes beyond just its current tools, so we included 18 criteria that assessed each vendor s ability to support customer implementations, its commitment to the ALM market, its ALM product strategy, and its corporate strategy. Market presence. We also took into account the size of the footprint each vendor has in the software market. Forrester s clients want to know how safe a bet it will be that a given ALM vendor will be in business several years from now in order to gauge how safely they can invest in that vendor s ALM tools. This year s ALM evaluation represents a departure from the 2010 Forrester Wave evaluating Agile development management solutions in three ways: 2 Opening the aperture to encompass all ALM scenarios. The 2010 evaluation focused exclusively on Agile support among ALM vendors. This year, we include both Agile and non- Agile use cases in the criteria.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 7 Treating Agile and Lean as a critical test of ALM offerings. At the same time, this year s evaluation by no means leaves Agile behind. Instead, we treat Agile and Lean as critical tests of an ALM vendor s offering. Incorporating systems engineering and embedded software. Given the increasing quantity of hybrid products those incorporating software into physical products ranging from cars to medical devices we have also included criteria covering software engineering outside of corporate IT and independent software vendors (ISVs). Evaluated Vendors Have Strong Tools And Substantial Market Presence Forrester included nine vendors in the assessment: Atlassian, CollabNet, HP, IBM, Microsoft, PTC, Rally Software, Rocket Aldon, and Serena Software. Each of these vendors has (see Figure 2): Depth and breadth of ALM offering. While there are many commercial and open source point solutions for particular activities (test management, build management, issue tracking, etc.), the title ALM vendor goes to companies that do more than provide a single capability (source control, defect tracking, etc.), however good it might be. We expect these tools to cover the majority of ALM use cases, as Forrester defines ALM. A broad ALM customer base. We also looked at the number of customers for a vendor s ALM products in 2011 (at least 750) as well as the growth in the vendor s customer base from 2010 to 2011. Definitive revenue from ALM. Finally, the criteria included the revenue from ALM. Once again, we included both the numbers for 2011 (at least $19 million) and the growth since 2010.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 8 Figure 2 Evaluated Vendors: Product Information And Selection Criteria Vendor Atlassian Product evaluated Atlassian OnDemand: JIRA GreenHopper Bonfire Confluence SharePoint Connector for Confluence Team Calendars for Confluence Bamboo FishEye Crucible Clover Crowd Bitbucket Atlassian IDE Connector for Eclipse Atlassian IDE Connector for IntelliJ IDEA Atlassian IDE Connector for Visual Studio 2008 and 2010 HipChat Stash Sourcetree JIRA Mobile Connect JIRA OnDemand GreenHopper OnDemand Bonfire OnDemand Confluence OnDemand Team Calendars for Confluence OnDemand Source OnDemand Bundle (FishEye, Crucible, and Subversion) Product version evaluated 5.0 5.10 2.2 4.2 1.5.3 2.3 4.1 2.7 2.7 3.1.6 2.4.2 3.5 or later 7 and later N/A 1.1 N/A Version release date June 2012 CollabNet TeamForge ScrumWorks Pro Subversion Edge Lab Management 6.2 6.0 3.0 2.5 June 2012 May 2012 April 2012 May 2012 HP HP Application Lifecycle Management (HP ALM) 11.5 11.5 June 2012 IBM Rational Collaborative Lifecycle Management Solution (Rational CLM): Rational Requirements Composer 4.0 Rational Team Concert 4.0 Rational Quality Manager 4.0 4.0 June 2012 Microsoft Microsoft Visual Studio 2010 May 2010 PTC Integrity 10.1 June 2012 Source: Forrester Research, Inc.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 9 Figure 2 Evaluated Vendors: Product Information And Selection Criteria (Cont.) Vendor Product evaluated Product version evaluated Version release date Rally Software Rally Community Edition Rally Enterprise Edition Rally Unlimited Edition Rally Quality Manager Rally Support Manager Rally Product Manager Rally Community Manager Rally Time Tracker Rally Idea Manager Rally Portfolio Manager Rocket Aldon Aldon Lifecycle Manager Aldon Deployment Manager Aldon Community Manager Aldon Report Manager Aldon Agile Manager 1.0 (e) 6.1/(i) 7.6 N/A 9.5.4 1.5 1.0 May 2011 N/A November 2010 June 2011 January 2011 Serena Software Development Manager Release Manager Requirements Manager Service Manager Demand Manager Request Center Dashboard ChangeMan ZMF Agile Planner 1.2 1.0 2.1 3.1 3.1 3.1 2.1 7.1 2.3 May 2012 Vendor selection criteria The vendor provides designated ALM product(s) and a stated ALM road map. The vendor has more than 750 customers. The vendor provides tools that cover the majority of ALM use cases, as Forrester defines ALM. The vendor generates more than $19 million in revenue. Source: Forrester Research, Inc.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 10 THE ALM FORRESTER WAVE Finds Many Leaders And Different Directions The vendors we evaluated for the ALM Forrester Wave provide a wide range of capabilities. Superficially, they look very much alike. Project management? Check. Issue tracking? Check. With such feature breadth, you might conclude that ALM is a mature market. But where are the vendors heading? The answer is, in very different directions. Software may be everywhere, but the form of software development and delivery varies widely across organizations. While this market now has many genuine leaders, they have reached this point only by identifying their core market. Therefore, the market is less mature than it might seem on the surface. When evaluating ALM vendors, it s important to consider not just the capabilities they provide but also the use cases for which they are optimized. Nowhere is this differentiation more clear than in the overarching principles that guide the evaluated vendors product strategies (see Figure 3): Leaders IBM and Microsoft provide broad ALM capabilities. Both IBM and Microsoft have maintained a strong portfolio of ALM tools designed to support common use cases for software development and delivery. The two companies both provide broad swaths of capabilities for common ALM activities, such as project definition and management, reporting, issue tracking, and task assignment. IBM and Microsoft come from a more general-purpose perspective on ALM, and both treat Agile practices as a special instance of ALM workflows, metadata, and reporting. Both companies support Agile as a template, a special configuration of their ALM tools. Note: For this Forrester Wave, Forrester scored Microsoft based on the functionality in the 2010 version its ALM tools. If scored today on the basis of its recent release of Visual Studio 2012, the results would be different. 3 Leaders PTC and Rally Software lead by focusing on specific use cases. In contrast to IBM and Microsoft, both Rally and PTC (originally MKS, before PTC s acquisition of the company in 2011) have succeeded in the ALM market by finding a niche in which to specialize for Rally, Agile, and for PTC, compliance. While this approach may limit their ability to support a broader swath of scenarios, their focus gives them the opportunity to provide thought leadership in their respective areas. Rally s focus on Agile and Lean, for example, has led the company to build top-notch planning capabilities into its tools. PTC s focus on compliance, including the highly regulated world of embedded software, has made it the only ALM vendor to provide out of the box many of the workflows and reports needed to satisfy regulatory authorities. Leaders CollabNet and Serena lead through delivering differentiated and broad capabilities. Both CollabNet and Serena have tailored their ALM offerings to more-general value propositions and needs. CollabNet sees its opportunity in the cloud, where ALM tools support
The Forrester Wave : Application Life-Cycle Management, Q4 2012 11 has been lagging behind IT s need to manage cloud development, testing, and deployment more effectively. Serena has recognized how important it is for organizations to effectively orchestrate the larger meta-process of software development and delivery, which spans the typical activities (coding, testing, building, etc.) that guide the development efforts of other commercial and open source tools. Strong Performers Atlassian and HP provide focused capabilities. Atlassian and HP show how far an ALM vendor can go with strong support for a piece of the overall life cycle. However capable individual tools such as JIRA and Quality Center may be, they are no substitute for a broader ALM offering. The two companies view this gap differently. Rather than defocus its product strategy, Atlassian prefers to provide a smaller range of strong capabilities than potentially do a mediocre job tackling a wider range of ALM challenges. This strategy makes it easier for Atlassian to deal with one of its biggest challenges, perception as an enterprise-ready vendor, without the added complexity of too many products and features. In contrast, HP has pushed hard to fill in the gaps in its ALM offering, moving far past Quality Center. The current version, ALM 11.5, provides a very robust set of tools that can compete with other ALM suites on a wide range of capabilities. Some of these new capabilities, such as the executive dashboard features, demonstrate that HP understands many tough ALM challenges and will invest in providing credible, differentiating features to address them. However, these tools are relatively new, and Forrester has not encountered many implementations of them yet. The next year will be critical for HP s ALM offering in gaining market traction. Contender Rocket Aldon is the operational dark horse of ALM. While most other ALM vendors strive to move down the value chain to the point of release and deployment, Rocket Aldon s product strategy starts with these later stages of the application life cycle. Automation plays a prominent role in Rocket Aldon s offerings, as do IT-governance-related features. The Rocket Aldon road map is, to some extent, extending upstream to cover scenarios (for example, product owner review of the backlog) that it currently does not address as thoroughly as production-related activities. This evaluation of the ALM market is intended to provide a starting point for your own evaluation. We encourage readers to view the detailed product evaluations and adapt the criteria weightings to fit their individual needs through the Forrester Wave Excel-based vendor comparison tool.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 12 Figure 3 Forrester Wave : Application Life Cycle Management Tools, Q4 2012 Strong Risky Strong Bets Contenders Performers Leaders PTC Microsoft Atlassian Serena HP IBM Rally CollabNet Go online to download the Forrester Wave tool for more detailed product evaluations, feature comparisons, and customizable rankings. Current offering Rocket Aldon Market presence Weak Weak Strategy Strong Source: Forrester Research, Inc.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 13 Figure 3 Forrester Wave : Application Life Cycle Management Tools, Q4 2012 (Cont.) Forrester s Weighting Atlassian CollabNet HP IBM Microsoft PTC Rally Software Rocket Aldon Serena Software CURRENT OFFERING Platform support Installing the product and getting going Administration Security Integration and customization Management Change elements Life-cycle traceability Reporting Analytics Running a project General collaboration capabilities Process configuration 50% 5% 5% 2% 3% 10% 5% 10% 15% 5% 10% 10% 10% 10% 3.47 3.40 4.80 4.00 2.20 4.80 3.95 2.47 3.63 2.75 1.38 4.75 4.70 2.24 3.51 4.10 4.40 4.75 2.60 4.90 3.89 3.74 3.26 3.70 2.28 4.30 2.90 2.36 3.53 4.30 4.20 3.80 4.00 3.96 3.85 3.85 3.36 2.90 2.80 1.18 4.38 3.80 4.00 4.60 3.79 4.48 4.38 4.60 4.30 4.80 3.20 3.91 3.90 3.60 4.20 4.75 3.96 4.74 3.47 4.30 3.20 4.30 4.20 2.56 4.01 1.50 3.60 4.00 4.75 4.42 4.21 3.64 4.65 3.70 4.25 3.31 3.95 3.20 4.60 3.40 3.97 4.75 4.08 4.55 4.28 4.15 3.80 1.26 1.97 0.10 2.82 1.50 1.05 2.06 1.44 0.66 1.80 2.44 3.10 1.05 4.45 3.62 3.10 2.80 3.75 2.20 4.14 3.38 3.38 4.15 4.26 3.25 3.15 3.61 STRATEGY Support for implementations Product strategy Corporate strategy Price Commitment 50% 15% 80% 5% 0% 0% 2.92 2.50 2.90 4.50 0.00 3.95 4.90 3.70 0.00 3.22 4.50 2.90 4.50 0.00 4.76 4.70 0.00 3.25 2.90 3.50 0.00 3.72 3.40 0.00 4.39 4.10 4.40 0.00 2.04 2.10 1.90 4.00 0.00 3.44 3.10 3.40 0.00 MARKET PRESENCE Installed base Financial strength Employees Support services Channel partnerships Global presence 0% 35% 5% 0% 35% 20% 5% 3.94 3.30 4.00 2.00 4.10 3.00 4.42 4.20 3.00 1.00 4.00 4.95 4.00 4.00 4.51 4.50 3.00 4.10 4.32 3.20 4.00 3.00 3.23 2.20 3.50 3.00 4.10 3.00 4.26 3.10 3.50 1.00 2.22 2.20 1.50 1.00 3.20 1.00 1.00 3.73 2.90 2.50 2.00 4.10 3.00 All scores are based on a scale of 0 (weak) to 5 (strong). Source: Forrester Research, Inc. ALM Vendors Have Yet To Fully Address Many Common Challenges While many aspects of the ALM tools market have improved substantially in the past two years since the Agile development management Forrester Wave evaluation, other areas remain ripe for innovation and improvement:
The Forrester Wave : Application Life-Cycle Management, Q4 2012 14 Testing. With one or two notable exceptions, the vendors we included in this study did not excel at any aspect of testing support. To some extent, the weakness of testing features is characteristic of the ALM tools market, not just the vendors we assessed. However, it s still worth noting that some specialty vendors have done a better job of connecting testing to other ALM activity. Some requirements tools vendors, for example, provide rapid test generation from requirements, something that many purveyors of bigger ALM suites have yet to implement. Collaboration. While we have seen some progress in this area, that s damning by faint praise considering how often collaboration is at the core of an organization s ALM challenges. Take, for example, the very common challenge of collaborating with an offshore partner. While some features, such as feeds or wikis, make it easier for onshore and offshore teams to eavesdrop on each other s activities, these are not vehicles for communicating product vision, mentoring people on the other team, or suggesting ideas for process improvement. Mobile support. At the risk of stating the obvious, many aspects of mobile development user experience, testing, deployment, etc. are significantly different from the elements involved in building middleware. Yet the typical ALM vendor today has done very little to address these needs, despite the continued expansion of the mobile market. Integration. ALM vendors have acknowledged the inevitability of integration with other commercial and open source tools. However, we re a long way from arriving at a common integration strategy that makes it easier for customers to select ALM tools based on their individual merits outside of integration. Some vendors provide an application programming interface (API) with few or no prebuilt connectors. Others provide plenty of connectors, with perhaps a weaker API for custom-built integrations. The fact that small companies, such as Kovair and Tasktop, have established themselves as third-party providers of ALM connectors is a testament to the weaknesses of the overall ALM market s integration efforts. VENDOR PROFILES Leaders IBM s offering is becoming more than the sum of its parts. IBM s position in this year s assessment of the ALM market is a testament to an important principle: It s not enough just to have good products the products, in combination, must support common activities in software development and delivery. In previous years, IBM has certainly provided strong solutions, such as Rational RequisitePro, Rational Team Concert, and Rational DOORS in specific functional areas such as requirements, planning, and project management. However, IBM provided multiple solutions in some of these areas, leading to confusion about what exactly each of these tools was supporting and how they all worked together.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 15 That confusion has largely dissipated with IBM s current ALM offering. Not only has IBM continued development of its strong suite of products, but it has also stitched them together in a more coherent way. The company has also made clearer the use cases it supports with its tools, such as Agile teams and embedded software development. Because of the company s strong product and portfolio strategy, it is possible to write about IBM s ALM offerings taken as a whole with far less irony. Rally Software continues its leadership in Agile/Lean. Rally targets a very healthy and growing opportunity: the expanding number of organizations that have adopted Agile and Lean. Rally s tools are optimized for Agile planning, project management, status reporting, and other actions that happen within and outside sprints. The company s acquisition of AgileZen, a Lean project management tool, was a natural fit for both Rally and its customers. So too was the addition of Rally Portfolio Manager, a tool for planning, decision-making, and management above the level of an individual project or product. Rally also continues to provide thought leadership in the Agile and Lean community. However, this focus on Agile has a downside: non-agile teams will find Rally s products and services far less attractive than other general-purpose ALM tools. Rally s leadership rests with its breadth and depth of capabilities for Agile teams, combined with a strong and focused corporate strategy. PTC continues its leadership addressing regulated and digital products needs. PTC provides strength where product-based development and compliance intersect. No other company, with the possible exception of IBM, has done as much to support teams of embedded software developers and systems engineers. PTC helps firms address digital product development needs for products such as appliances, vehicles, and medical devices that have software components while still maintaining support for teams building and delivering software-only products and projects. PTC s strength rests on its long-standing support for addressing governance and compliance challenges. Burrowing into this niche has its concomitant cost: a more complex product that is harder to justify adopting for teams that don t have to satisfy the needs of auditors or government regulators. CollabNet provides a flexible ALM framework optimized for the cloud. As already noted, one of the strengths of CollabNet s ALM suite is the flexibility with which users can define ALM content. We also see this flexibility pay off when trying to effectively address the needs of complex teams or complex projects, where there is less chance for a one-size-fits-all design. CollabNet has also invested heavily in supporting cloud development and deployment. Not only are CollabNet s tools available in the cloud (CloudForge), but they also include features optimized for developing, testing, and deploying cloud systems.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 16 Microsoft shortens cycle times. Starting with Visual Studio at the front end and through to Team Foundation Server (TFS) at the back end, Microsoft s ALM portfolio provides broad life-cycle support. Microsoft s overarching goal is to shorten cycle times, which requires a combination of individual measures (task management, test automation, etc.). Microsoft deserves recognition for understanding how central collaboration is to ALM. While the M word, management, is part of the ALM acronym, ALM initiatives frequently fail because they emphasize control over collaboration. Features such as visualization via PowerPoint and translating recorded manual testing into automated tests emphasize collaboration in these two examples, with business users. Serena Software orchestrates IT projects from idea to deployment. Since retooling its ALM strategy, Serena has made impressive advances at both the individual tool and the suite level. Central to its product strategy is orchestration, a vision of ALM as an ongoing, rhythmic activity. Not surprisingly, given this vision of a regular flow of activity (conceive, build, test, deploy), Serena has one of the best workflow designers available. Serena s recent enhancements to its ALM portfolio, such as the Serena Release Manager, provide more support in this end-toend orchestration model. Strong Performers For HP, quality is truly central, but other ALM capabilities are growing. HP Quality Center is the fulcrum on which HP s ALM strategy rests. HP Quality Center is a ubiquitous part of many organizations ALM infrastructure. Consequently, other ALM vendors integrate with HP Quality Center as a matter of course, almost as a price of entry into any potential customer s shortlist. HP has improved upon and expanded its ALM capabilities beyond test management and test automation in the two years since the previous Forrester Wave evaluation of this space. HP now provides improved requirements management and traceability. Another example, HP s Application Lifecycle Intelligence (ALI), offers a broad set of ALM data aggregation and analysis features, with strong reporting and dashboarding components. Given the importance of this information for everything from team-level continuous improvement to executivelevel portfolio management, it represents one of many areas where HP has moved quickly to distinguish itself as a serious ALM player. Atlassian focuses on issue tracking and collaboration within teams. Atlassian s JIRA, a widely popular issue tracking tool, often serves double duty as a requirements management system. The company s Confluence Wiki is equally popular with development teams. The two products, in tandem, support the various kinds of day-to-day work done within software development teams, which are always looking for ways to better collaborate (for example, through a wiki).
The Forrester Wave : Application Life-Cycle Management, Q4 2012 17 While Atlassian has decided to provide a narrower set of ALM tools than other vendors, it continues to hone the capabilities in the areas where the company believes it can excel, such as providing a marketplace for its developer community to provide plug-ins. The company has made moves into more enterprise-level scenarios, with new capabilities such as Stash, a tool for the management of Git repositories that enterprise customers demand. Atlassian has also made enhancements to its support, sales force, and other aspects of its business needed to gain ground in the enterprise market. Contenders Rocket Aldon provides strong operational support. Unlike other ALM vendors, Rocket Aldon s roots lie in software delivery, not software development. Rocket Aldon has focused from the beginning on the operational side, including both automation and reporting. Unlike other vendors, Rocket Aldon is expanding backward, not forward, in the software timeline, adding support for more upstream ALM activities, such as demand management and requirements. Supplemental MATERIAL Online Resource The online version of Figure 3 is an Excel-based vendor comparison tool that provides detailed product evaluations and customizable rankings. Data Sources Used In This Forrester Wave Forrester used a combination of three data sources to assess the strengths and weaknesses of each solution: Hands-on lab evaluations. Vendors spent one day with a team of analysts who performed a hands-on evaluation of the product using a scenario-based testing methodology. We evaluated each product using the same scenario(s), creating a level playing field by evaluating every product on the same criteria. Vendor surveys. Forrester surveyed vendors on their capabilities as they relate to the evaluation criteria. Once we analyzed the completed vendor surveys, we conducted vendor calls to clarify and confirm their products capabilities. Customer reference calls. To validate product and vendor qualifications, Forrester also conducted reference calls with two of each vendor s current customers.
The Forrester Wave : Application Life-Cycle Management, Q4 2012 18 The Forrester Wave Methodology We conduct primary research to develop a list of vendors that meet our criteria to be evaluated in this market. From that initial pool of vendors, we then narrow our final list. We choose these vendors based on: 1) product fit; 2) customer success; and 3) Forrester client demand. We eliminate vendors that have limited customer references and products that don t fit the scope of our evaluation. After examining past research, user need assessments, and vendor and expert interviews, we develop the initial evaluation criteria. To evaluate the vendors and their products against our set of criteria, we gather details of product qualifications through a combination of lab evaluations, questionnaires, demos, and/or discussions with client references. We send evaluations to the vendors for their review, and we adjust the evaluations to provide the most accurate view of vendor offerings and strategies. We set default weightings to reflect our analysis of the needs of large user companies and/or other scenarios as outlined in the Forrester Wave document and then score the vendors based on a clearly defined scale. These default weightings are intended only as a starting point, and we encourage readers to adapt the weightings to fit their individual needs through the Excel-based tool. The final scores generate the graphical depiction of the market based on current offering, strategy, and market presence. Forrester intends to update vendor evaluations regularly as product capabilities and vendor strategies evolve. Endnotes 1 Forrester s research indicates that the dependence on software to achieve business outcomes will only continue through the next decade. Therefore, while packaged applications will continue to be important, software development and delivery for building new applications, customizing packaged systems, or performing custom integrations will be critical for bridging the gap between software capabilities and business needs. See the January 30, 2012, BT 2020: To Thrive In The Empowered Era, You ll Need Software, Software Everywhere report. 2 The previous Forrester Wave contained many similar questions but focused more narrowly on Agile team support. See the May 5, 2010, The Forrester Wave : Agile Development Management Tools, Q2 2010 report. 3 The Forrester Wave methodology uses a cutoff date for vendor features features added after that date are excluded from scoring even though they may reach the market before the Forrester Wave publishes. This evaluation used July 1, 2012, as its cutoff, which means that the scoring in this Forrester Wave is based on the 2010 version of Microsoft s ALM tools. Readers should note that Microsoft released the 2012 version on September 12, 2012, and that it includes some significant enhancements in areas such as testing, Kanban, lab management, and requirements visualization. Using today s available features, Microsoft s scoring would be different.
About Forrester A global research and advisory firm, Forrester inspires leaders, informs better decisions, and helps the world s top companies turn the complexity of change into business advantage. Our researchbased insight and objective advice enable IT professionals to lead more successfully within IT and extend their impact beyond the traditional IT organization. Tailored to your individual role, our resources allow you to focus on important business issues margin, speed, growth first, technology second. for more information To find out how Forrester Research can help you be successful every day, please contact the office nearest you, or visit us at www.forrester.com. For a complete list of worldwide locations, visit www.forrester.com/about. Client support For information on hard-copy or electronic reprints, please contact Client Support at +1 866.367.7378, +1 617.613.5730, or clientsupport@forrester.com. We offer quantity discounts and special pricing for academic and nonprofit institutions. Forrester Focuses On Application Development & Delivery Professionals «Responsible for leading development and delivery of applications that support your company s business strategies, you also choose technology and architecture while managing people, skills, practices, and organization to maximize value. Forrester s subject-matter expertise and deep understanding of your role will help you create forward-thinking strategies; weigh opportunity against risk; justify decisions; and optimize your individual, team, and corporate performance. Andrea Davies, client persona representing Market Insights Professionals Forrester Research, Inc. (Nasdaq: FORR) is an independent research company that provides pragmatic and forward-thinking advice to global leaders in business and technology. Forrester works with professionals in 17 key roles at major companies providing proprietary research, customer insight, consulting, events, and peer-to-peer executive programs. For more than 29 years, Forrester has been making IT, marketing, and technology industry leaders successful every day. For more information, visit www.forrester.com. 60080