Sales. Guide. for Channel Partners

Transcription

1 Sales Guide for Channel Partners

2 Table of Contents 2015 SmartBear Confidential TABLE OF CONTENTS SECTION I: How To Use This Guide...3 SECTION II: Who is SmartBear?...4 The Leader in Software Quality Tools for the Connected World...4 SmartBear Products...5 SECTION III: SmartBear Value Guides...7 SmartBear Development Tools...7 Collaborator Discovery Guide...7 Collaborator Differentiator...11 AQTime Discovery Guide...14 SmartBear Test Tools...16 Test Discovery Guide...16 Test Differentiator...21 API Readiness Tools...25 Ready! API Discovery Guide...25 Ready! API Differentiator...29 ServiceV Pro Discovery Guide...33 ServiceV Pro Differentiator...36 SmartBear s Monitoring Tools...39 AlertSite UXM Discovery Guide...39 AlertSite UXM Differentiator...45 SECTION IV: SmartBear s Channel Management Team

3 2015 SmartBear Confidential How to Use This Guide SECTION I: HOW TO USE THIS GUIDE Dear Partners, This sales guide has been prepared for use by Partners and is a CookBook for both Sales personnel and Consultants (if you are a service Partner) on how to position SmartBear s products in your customer base. Section I: Who is SmartBear? Resources: There are two diagrams to illustrate the SmartBear Tools portfolio. The first diagram displays where each tool fits in the software lifecycle. The next diagram is a quick reference guide and the Products and Solutions overview is a helpful summary of core capability of each of the Smart- Bear tools and who the target users and also buyers of the tool will be. Section II: SmartBear Value Guides Each SmartBear tool set also has a Value Discovery Guide which comes in two parts: Discovery Guide is a positioning document, which makes no reference to any of our tools, but helps draw out the pain points the clients may be experiencing and allow the Salesperson or Consultant understand which tool(s) best fit the customers need and also assess what additional services / solutions they may require. Differentiator Guide is a comparative document showcasing the feature set of each SmartBear tool and how it can solve the client s pain points. It also includes sample references from existing client deployments to support the argument for choosing SmartBear tools, as well as a summary of SmartBear tool strengths & weaknesses against other competitor offerings. Section III: SmartBear s Channel Management Team details on your Channel Management Team. Sincerely, The SmartBear Partner Team 3

4 Who is SmartBear? 2015 SmartBear Confidential SECTION II: WHO IS SMARTBEAR? The Leader in Software Quality Tools for the Connected World Nearly all applications today are connected to other applications through APIs, to mobile devices, and the cloud. The Connected World drives tremendous value for application providers, but introduces significant challenges to effectively build, test, and monitor the applications and the connections between them. SmartBear helps companies build and deliver the world s greatest applications through collaborative development, functional testing, performance testing and performance monitoring across API, mobile, web, and desktop. 4

5 2015 SmartBear Confidential SmartBear Products SMARTBEAR PRODUCTS SmartBear Products What the Product does Business Value Value Add Partner Products Additional Business Value Offered by Partner Product Add-On SmartBear and/or Partner Products that combine to create Solutions Used By Development Manages Document and Code Reviews. Integrates with 16 industry leading Software Configuration Management Tools. Integrates with Eclipse & Visual Studio IDEs. Supports multiple document formats. Code Performance Profiling (C, C++,.Net, Java, Silverlight, Delphi, VBScript, JavaScript). Memory profiling (allocation & leaks). Code coverage analysis (% tested vs used). Improved governance. Better quality software shipped. Demonstrate compliance to Regulatory and Industry Standards. Improved user experience (performance /response times). Better quality software shipped. SCM & IDE Integrations with: AccuRev, CMVC, CVS, Git, Mercurial, MKS Integrity, Perforce, Rational ClearCase Rational Synergy, Rational Team Concert, Subversion, StarTeam, Surround SCM, TFS, Vault, VSS. Integrated Development Environment. Whole Project Team. Developers Test Automation Eng Testing Manage test cases in central location. Manage test hosts and schedules. Create and manage defects. Trace Tests to User Stories. Report on test execution. Improved governance / traceability. Better quality software shipped. Demonstrate compliance to Regulatory and Industry Standards. OpsHubs Integrators including: AccuRev, Jira, Bugzilla, CVS, Git, HP ALM, HP Quality Center, Rational ClearCase, ClearQuest, Team Concert, TFS, Rally, TestTrack, Serena Business Manager, Subversion, Trac. Integrated Development Environment. With TestComplete & OpsHub integrators. Test Manager Testers Test Automation Eng Test automation tool. Supports Desktop, Mobile & Web testing. Test Recording and Playback. Supports scripting using VB- Script, JScript. Integrates with Agile and CI tools. Mobile support Native apps Hybrid apps Xamarin apps Faster time to market. Better quality software shipped. Test Automation Frameworks(TAF) (supplied by various Partners) Faster time to market Increased productivity With QAComplete (Solution: Provides integrated test management and automated testing). Test Manager Testers Test Automation Eng 5

6 SmartBear Products 2015 SmartBear Confidential SMARTBEAR PRODUCTS Web load testing tool. Supports Cloud, RIA and Mobile load testing (Ajax, Flex, ASP.Net & Silverlight). Automates Web load testing. Comprehensive Load Test reports. Improved user experience (performance / response times). Better quality software shipped. Performance Engineer Testers API Readiness API functional testing tool. Supports SOAP, REST, HTTP, JMS, AMF, JDBC. Supports Virtualization & API Discovery. Plugin architecture allows for extensions. API Readiness Platform - SoapUI NG Pro, LoadUI NG Pro, ServiceV Pro. Better quality software shipped. Apiary - API Blueprint IBM - API Manager MuleSoft - RAML Reverb Technologies - Swagger. Full API documentation. Allows users to browse Mule- Soft s APIHub directory for APIs using RAML or Swagger definitions. With LoadUI Pro (Solution: Provides API functional & load testing). Testers Test Automation Eng API load testing tool. Supports distributed API load testing. Automates API load testing. Comprehensive Load Test reports. Improved user experience (performance / response times). Better quality software shipped. With SoapUI Pro (Solution: Provides API functional & load testing). Performance Engineer Testers User Experience / Application Performance Monitoring Website speed & availability monitoring. Mobile Monitoring. API monitoring. SLA monitoring. Transaction tracing. Improved user experience (performance / response times / outage reduction). Better quality software shipped. Faster time to find / isolate code defects & performance hotspots. Perfecto Mobile. With SoapUI Pro & LoadUI Pro (Solution: Provides End to End Web, application performance monitoring and API functional & load testing) Perfecto Mobile (Solution: Users will be able to monitor their applications across real devices in the Perfecto Mobile cloud, using preexisting scripts) IT Manager CIO NOC Manager 6

7 2015 SmartBear Confidential Collaborator Discovery Guide SECTION III: SMARTBEAR VALUE GUIDES SmartBear Development Tools Collaborator Discovery Guide Value Driver Improve Code Design and Quality Improve Agility through Collaboration Across Functional Teams and Locations Generate Metrics to Improve Process and Verify Compliance Accelerate Team Member Onboarding Discovery Questions (Current State) Describe how successful your team has been at finding defects during development (rather than QA or the customer). Describe how your distributed team members collaborate. Describe what you are currently doing to ensure that each developer has some understanding of the overall code base. If someone were to leave, what does that mean for your team? Describe how you prepare for a compliance audit. Tell me about the development tools you use. Tell me how product management, development, and QA collaborate on user stories, code, and test plans and ensure they stay in sync. Describe how your distributed team members collaborate. Describe what you are currently doing to ensure that each developer has some understanding of the overall code base. If someone were to leave what does that mean for your team? Describe for me how you prepare for a compliance audit. What information is typically necessary when preparing for an audit? How do you locate the materials pertinent to an audit? Describe what you are currently doing to ensure that each developer has some understanding of the overall code base. Tell me about the process you use to get new members of the development team up to speed on their responsibilities. If a developer on your team leaves, describe how you ensure that the rest of the team can take over his or her work. Before Scenarios Defects found by QA/customer rather than development. Lack of collaboration between members of the development organization particularly those away from HQ. Lack of understanding of the overall code base. Tedious manual effort to document comp-liance. Multiple standalone tools for development. Not agile due to lack of communication between product management, deve-lopment and QA. Lack of collaboration between members of the development organization particularly those away from HQ. Lack of understanding of the overall code base. Tedious manual effort to document compliance. Materials spread out across multiple systems, don t talk to each other. Lack of understanding about what s needed and increased time spent finding the appropriate materials. Developers know their own code, but not what others are working on. Team members must learn on their own rather than from each other. When a developer leaves, the other developers have little idea what that developer was wor-king on and how to pick up his or her work. 7

8 Collaborator Discovery Guide 2015 SmartBear Confidential Value Driver Improve Code Design and Quality Improve Agility through Collaboration Across Functional Teams and Locations Generate Metrics to Improve Process and Verify Compliance Accelerate Team Member Onboarding Negative Consequences High re-work costs and damage to reputation. Inability to quickly respond to changing customer and market dynamics. Inefficient development process (higher cost). Intellectual capital rests with individuals rather than team. High cost of preparation efforts and potential fines. Inability to quickly respond to changing customer and market dynamics. Inefficient development process (higher cost). Intellectual capital rests with individuals rather than team. Slower, error prone development. High cost of preparation efforts and potential fines. Slower, error prone development. Increased costs due to time spent finding the appropriate materials. Inefficient development process (higher cost). Intellectual capital rests with individuals rather than team. Slower, error prone development. Slower, error prone development. Discovery Questions (Future State) Describe the role you would like developers to play in driving quality. When market dynamics change, tell me about how you would like your extended development team to be made aware and respond. Describe how your teams could work together more effectively. How would capturing the knowledge of each developer help you? Describe how you might be able to access information more quickly. Explain how your development environment could be improved. Describe the role you would like developers to play in driving quality. When market dynamics change, tell me about how you would like your extended development team to be made aware and respond. Describe how your teams could work together more effectively. How would capturing the knowledge of each developer help you? Explain how your development environment could be improved. Describe how you might be able to access information more quickly. Explain how your development environment could be improved. Describe how you might prove compliance more easily. When market dynamics change, tell me about how you would like your extended development team to be made aware and respond. Describe how your teams could work together more effectively. How would capturing the knowledge of each developer help you? Explain how your development environment could be improved. More defects found by development, before QA and the customer. More defects found by development, before QA and the customer. Easily document and report on compliance. Extended development team collaborates and reacts to change. After Scenarios Extended development team collaborates and reacts to change. Greater collaboration among developers and sharing of best practices. Greater knowledge of the overall code base among developers. Easily document and report on compliance metrics. Development tools used all talk to each other. Extended development team collaborates and reacts to change. Greater collaboration among developers and sharing of best practices. Greater knowledge of the overall code base among developers. Development tools used all talk to each other. Development tools used all talk to each other. Greater collaboration among deve-lopers and sharing of best practices. Greater knowledge of the overall code base among developers. Development tools used all talk to each other. 8

9 2015 SmartBear Confidential Collaborator Discovery Guide Value Driver Improve Code Design and Quality Improve Agility through Collaboration Across Functional Teams and Locations Generate Metrics to Improve Process and Verify Compliance Accelerate Team Member Onboarding Positive Business Outcomes Cost savings through reduction of rework. Faster, more nimble development. Faster onboarding, greater ability to address changes to code base. Minimize risk due to resource departure. Cost savings through reduction of rework. Faster, more nimble development. Faster onboarding, greater ability to address changes to code base. Minimize risk due to resource departure. Cost savings through reduction of rework. Faster, more nimble development. Quickly respond to audit requests. Cost savings through reduction of rework. Faster, more nimble development. Faster onboarding, greater ability to address changes to code base. Quickly respond to audit requests. How would you enable developers to find more defects? How would you enable developers to find more defects? How would you enable developers to find more defects? How would you enable developers to find more defects? Discovery Questions (What s Required to get there?) Describe how you would make the process of collaboration easier. How do you ensure that collaboration happens among developers? How would you facilitate that knowledge transfer? What would the system look like? Describe the integrations you would like your peer review tool to have. Describe how you would make the process of collaboration easier. How do you ensure that collaboration happens among developers? Describe how you would make the process of collaboration easier. How do you ensure that collaboration happens among developers? How would you facilitate that knowledge transfer? What would the system look like? Describe how you would make the process of collaboration easier. How do you ensure that collaboration happens among developers? How would you facilitate that knowledge transfer? Required Capabilities Efficient means to review code and track fin-dings. Central location to manage documents and facilitate peer code review. Forum for developers to share code and insight on code. Central location for reporting on code comp-liance. Ability to integrate with SCMs and IDEs currently in use. Efficient means to review code and track findings. Central location to manage documents and facilitate peer code review. Efficient means to review code and track findings. Central location to manage documents and facilitate peer code review. Forum for developers to share code and insight on code. Central location for reporting on code compliance. Efficient means to review code and track findings. Central location to manage documents and facilitate peer code review. Forum for developers to share code and insight on code. Metrics # defects found by development and defect density. Adoption rate and delivery improvements. Speed of delivery. # of participants in each review, overlapping reviewers. # of reporting elements around accountability. Speed of delivery. Speed to productivity (for new hires). Decrease in rework. # defects found by development and defect density. Speed of delivery. # of participants in each review, overlapping reviewers. # of reporting elements around accountability. Speed of delivery. Speed to productivity (for new hires). # of participants in each review, overlapping reviewers. 9

10 Collaborator Discovery Guide 2015 SmartBear Confidential Value Driver Improve Code Design and Quality Improve Agility through Collaboration Across Functional Teams and Locations Generate Metrics to Improve Process and Verify Compliance Accelerate Team Member Onboarding Peer Review Peer Review Peer Review Peer Review How We Do It (Satisfy the Required Capabilities) Integration with SCM/IDE. Easily upload materials. Invite the appropriate people to review and capture their conversations. Defect tracking. Metrics and Reporting Easily upload materials. Invite the appropriate people to review and capture their conversations. Defect tracking. Invite the appropriate people to review and capture their conversations. Defect tracking Metrics and Reporting Capture the appropriate metrics and render them in reports. Invite the appropriate people to review and capture their conversations. Defect tracking. Capture the appropriate metrics and render them in reports. Peer Review Peer Review Peer Review Peer Review How We Do It Better Integration with 11 SCMs and the major IDEs. Create users and review groups, leveraging existing structures like LDAP. Synchronous and asynchronous conversations tied to places in materials. Structured workflow to ensure resolution of identified defects. Create users and review groups, leveraging existing structures like LDAP. Synchronous and asynchronous conversations tied to places in materials. Structured workflow to ensure resolution of identified defects. Adaptive workflow, templates and fields. Synchronous and asynchronous conversations tied to places in materials. Structured workflow to ensure resolution of identified defects. Adaptive workflow, templates and fields. Metrics and Reporting Synchronous and asynchronous conversations tied to places in materials. Structured workflow to ensure resolution of identified defects. Adaptive workflow, templates and fields. Adaptive workflow, templates and fields. Metrics and Reporting Automatically gather appropriate. review metrics for reporting. Automatically gather appropriate review metrics for reporting. Electronic signature. Customizable reports. Customizable reports. Proof Points Cisco Problem: Inefficient manual review process Solution: Tool assisted code review Results: More defects found, lower cost of development (less rework), faster onboarding. Intuit Problem: Needed to make it easy to promote the best practice of reviews and inspections, while capturing valuable process data Solution: Implemented Collaborator to track metrics and drive best practices Results: Cost savings through reduction of rework and faster, more nimble development. USAA Problem: Could not be Agile because they are worldwide and could not collaborate effectively Solution: Used Collaborator across development, product management, and testing teams inserting user story, code, and test plan into a single review Results: Increased collaboration across extended development team, now consider themselves much more Agile. Intuit Problem: Needed to make it easy to promote the best practice of reviews and inspections, while capturing valuable process data Solution: Implemented Collaborator to track metrics and drive best practices Results: Cost savings through reduction of rework and faster, more nimble development. Intuit Problem: Needed to make it easy to promote the best practice of reviews and inspections, while capturing valuable process data Solution: Implemented Collaborator to track metrics and drive best practices Results: Cost savings through reduction of rework and faster, more nimble development. 10

11 2015 SmartBear Confidential Collaborator Discovery Guide Value Driver Improve Code Design and Quality Improve Agility through Collaboration Across Functional Teams and Locations Generate Metrics to Improve Process and Verify Compliance Accelerate Team Member Onboarding Proof Points GE Healthcare Problem: Proving compliance Solution: Implemented Collaborator for reporting and electronic signatures for accountability Results: Responded more quickly to audit requests. HP Problem: Measure return of investment on peer review Solution: Initiate and track peer review costs and time savings Results: Return on investment is 10-to-1 and saved $21.4 million per year. Intuit Problem: Needed to make it easy to promote the best practice of reviews and inspections, while capturing valuable process data Solution: Implemented Collaborator to track metrics and drive best practices Results: Cost savings through reduction of rework and faster, more nimble development. GE Healthcare Problem: Challenge with proving compliance Solution: Implemented Collaborator for reporting and electronic signatures for accoun-tability Results: Responded more quickly to audit requests. HP Problem: Needed to measure return of investment on peer review Solution: Initiate and track peer review costs and time savings Results: Return on investment is 10-to-1 and saved $21.4 million per year. HP Problem: Needed to measure return of investment on peer review Solution: Initiate and track peer review costs and time savings Results: Return on investment is 10-to-1 and saved $21.4 million per year. Collaborator Differentiator Type (U,C,H) (U,C,H) Differentiator Customer Value Defensibility Qualifications Unique Peer Review Code and Documents User stories. Test plans. Code. Revenue Ability to speed development and react to changes in specs. Decrease rework time. USAA Solution: Use Collaborator for document review Results: A faster, more nimble Agile team that can react to changing customer information (user stories). IBM Solution: Peer review Results: Each hour of peer review during development saved 20 hours of testing and 82 hours of rework. How is the extended development team Agile? Describe what is preventing them from being more Agile. How is the development team made aware of changes to user stories after initiating a project? Tell me about how you review test plans and user stories today. Collaboration across extended development team reduces lack of knowledge transfer between teams. Comparative Customized Workflow Custom fields (participants, review). Customizable review templates. Groups/review pools. Quicker, more efficient reviews. Eliminate the need to change current process. Intel Solution: Custom fields and templates Results: By allowing/forcing users to enter a value for time spent on code prior to review, Intel saved time in calculating this metric and reduced risk by handling all time metrics with one tool. Explain the procedures that are unique to your company that could prevent you from collaborating through peer review. What specific metrics do you want to capture during reviews? Describe how you manage different review types across projects and groups. 11

12 Collaborator Differentiator 2015 SmartBear Confidential Type (U,C,H) (U,C,H) Differentiator Customer Value Defensibility Qualifications Comparative Metrics Collection LOC reviewed. Defects found. Time spent in review. Revenue Increase speed of development. Optimize the team s ability to find bugs in development. Ability to determine the correct processes and allocation of resources. ebay Solution: Utilizing metrics and reporting Results: ebay is reviewing metrics and making changes based on them, which has increased speed and quality, and reduced risk. What metrics do you collect today around code review? What data do you use to make process changes? Tell me about the amount of time you spend in code review today and how you can confirm it s optimal. Comparative Scalability # of clients accessing the server simultaneously. Ability to handle multiple locations. Revenue Quickly ramp up development community. Scale to the enterprise on one server. Proven enterprise solution. Alcatel Lucent Solution: Single global solution for peer review Results: Running 800 concurrent users across the globe on a single instance of Collaborator. Describe how you can ensure the solution you choose can scale to your current and future enterprise requirements. Tell me about how you are currently supporting globally dispersed teams. How do you ensure that your review data is easily accessible anywhere in the world? Comparative Multiple Integrations # of SCMs. # of IDEs. No need for additional review systems. KLS Tencor Solution: A peer review system that could integrate with their multiple SCMs and IDEs Results: Integration with multiple SCMs and IDE (Git and RTC) allowed them to get up to speed quickly. How are you able to support multiple SCMs and IDEs? How would the peer review system you choose support multiple SCMs and IDEs? Remove manual steps and human intervention. Holistic Best in Class Support Phone support. Utilize development as third tier support. Revenue Customers get up to speed quickly and can maintain their environment. Support mitigates higher price paid for product. GE Healthcare Solution: Looking for new best practices and support assistance Results: Acted as trusted advisor, got them up to speed quickly, helped manage their integrations. Tell me about how you are going to manage the installation and maintenance of your peer review system. How readily available is the support from other vendors that you re considering? Customers get up to speed quickly and can maintain their environment. 12

13 2015 SmartBear Confidential Collaborator Differentiator Competitor SmartBear Strengths and Weaknesses How They Compete Against Us Our Strategic Response Crucible Gerrit Customizable workflows. Metrics/reporting. UI. Lower price up to 5 users. Integration with Jira. Good enough. - Graphical reports. + Customizable workflows. Free tool. + Reporting/metrics. Google lineage. - Price. Good enough. - Lack of Eclipse/Git Integration. You get what you pay for (i.e.: customization, integrations) and we offer strong support. We offer a limited integration with Jira in addition to many other far more important features. You get what you pay for (i.e.: customization, integrations) and we offer strong support. Long-term cost of unsupported free tools. Customer Objection Our Response Proof Points Price Our price is cheap relative to the value of document review, metrics and reporting, integrations. Schlumberger Problem: Know they want code review but were having difficulty justifying the price Solution: Selling Collaborator internally by positioning it as a necessary collaboration tool across their three main geographies. Code review takes too much time The time is worth the value gained from a better code base. Kronos Problem: Did not believe code review would be worth it Solution: Collaborator Result: Post-implementation, they found code review to be a worthwhile investment with a strong ROI. Code review is not Agile (too rigid) Code review helps enterprises become more Agile. USAA Problem: Wanted to be Agile but could not, given their size and distributed nature of the teams Solution: Collaborating through peer review Result: Enabled the teams to be more nimble and more aware of changes during the development process. Not a priority Go higher in organization in order to learn more about the customer s value drivers. Schlumberger Problem: Know they want code review but it s not a priority for the overall company Solution: Schlumberger UK has a major advocate for peer review that is selling the product first in his location, but wants to eventually sell it throughout corporate (UK, Houston, France). Other solutions are good enough Have you considered the total cost of ownership for open source? PlantCML Problem: They were looking for a way to eliminate the manual steps of a code review and gather metrics Solution: Collaborator Result: Eliminated many of the manual steps in their code review process and allowed them to generate custom reports to gather specific types of metrics. 13

14 AQTime Discovery Guide 2015 SmartBear Confidential AQTime Discovery Guide Value Driver Discovery Questions (Current State) Improve Application Performance What impact do performance bottlenecks within your applications have on end user satisfaction? How do you assess your application for possible memory leaks? How do you know what the resource allocation looks like for your applications? How do you ensure adequate code coverage when testing for memory leaks? Before Scenario Negative Consequence No tools used to find performance bottlenecks in applications. No ability to do memory profiling to identify sources of leaks. New applications are released that perform inefficiently. Applications are unstable and overly prone to crash. Developers can t quickly assess applications to understand how efficiently they use resources. New applications released without knowing whether code coverage is sufficient to look at all parts of the application. Discovery Questions (Future State) How would you like to go about detecting memory leaks and getting more visibility into resource consumption of your applications? What do you need to be more confident that new application releases will be as stable as possible and perform to end user expectations? How valuable would it be to know whether tests cover the entire application and whether additional tests are required? After Scenario Applications are designed to be as efficient as possible in consumption of memory and other computing resources. Confidence that applications will be stable and less likely to crash. Realistic tests that accurately simulate users and activities, getting as close to real life as possible. Positive Business Outcome Better end user experience and higher satisfaction. Applications work according to original business goals. Discovery Questions (What s Required to get there?) What is necessary for you to quickly find performance bottlenecks in your applications and scripts? How much more confidence would you have in your applications if you could detect different types of memory and resource leaks? How would you like to be able to perform code coverage analysis to understand whether tests are looking at every part of the application? 14

15 2015 SmartBear Confidential AQTime Discovery Guide Value Driver Improve Application Performance Required Capabilities Accomplish performance profiling of the entire or specific parts of the application. Find and monitor memory leaks. Ability to do continual code coverage analysis to assess which parts of the application are fully tested, partially tested, or not tested at all. Metrics Memory consumption rates and performance times. Total memory usage, memory consumption by category, and resource types. Percent coverage rates and lines of code covered or uncovered. Performance hot spots. Function level and line level hit counts and memory allocations. How We Do It (Satisfy the Required Capabilities) How We Do It Better Find performance bottlenecks in 32-bit and 64-bit C/C++, Delphi,.NET, Java, and Silverlight applications and scripts. Start with a quick performance inspection using the lightweight Sampling profiler and then drill deeper into the hot spots using the more accurate Performance profiler. See application s memory and resource allocations in real time to detect excessive memory and resource usage. Monitor total memory usage, memory usage by category (heap, reserved virtual memory etc). AQtime provides code coverage analysis for C/C++,.NET, Silverlight, Java and Delphi applications. AQtime can be used to measure code coverage during manual application runs, as well as integrated into your automated regression tests for continuous coverage analysis. Code coverage analysis shows which source files, functions and lines of code have been covered by tests, are untested or only partially tested. This helps you know what additional tests to create and how to improve your existing tests. Comprehensive yet easy-to-understand reports that allow developers to easily locate performance bottlenecks, memory and resource leaks and other issues in applications, and optimize the source code appropriately. Summary and detailed reports with ability to visualize critical paths. Use as a standalone performance profiler, or integrated into Microsoft Visual Studio and Embarcadero RAD Studio IDEs for improved productivity. Focus on specific parts of source code and select the exact source files, classes or methods to performance profile or exclude from profiling; start and stop profiling any time during the application run or start and stop profiling automatically when the application execution enters or exits a specific method. Compare profiling results of different application runs to easily view performance difference before and after optimizations; view the difference as exact values or as a percentage. Merge code coverage results from multiple test runs to get aggregate coverage statistics for your entire project. This is useful for large projects where testing is performed in multiple environments. AQtime can also merge results automatically after each coverage session to save you time and allow you to focus on the end results. AQtime includes a Failure Emulator to simulate faults during your application run and monitor their impact on the application behavior to evaluate the effectiveness of application s built-in fault detection and recovery code, determine the performance loss due to failures, and identify dependability bottlenecks that may need to be addressed using fallback solutions. Work smarter and optimize your code in Microsoft Visual Studio or Embarcadero RAD Studio. Find performance issues, memory and resource leaks right from your IDE. 15

16 Test Discovery Guide 2015 SmartBear Confidential SmartBear Test Tools Test Discovery Guide Value Driver Improve Quality Across Mobile, Web and Desktop Increase Speed to Delivery Ensure Predictable Performance Improve Flexibility and Ease of Use Across All Skill Sets Discovery Questions (Current State) What is the impact of doing mostly manual testing? Walk me through how your tools work together. Describe how well your current tools adapt if development is releasing new product initiatives (i.e., web to mobile migration). Explain how your testing is affected when new versions of supported technologies are introduced. What is the impact of doing mostly manual testing? Walk me through how your tools work together. Describe how well your current tools adapt if deve-lopment is releasing new product initiatives (i.e., web to mobile migration). Explain how your testing is affected when new versions of supported technologies get introduced. Explain how you on-board a team member and how long it takes. Walk me through how your tools work together. What is the impact of doing mostly manual testing? Explain how you on-board a team member and how long it takes. How do you ensure team members with limited scripting abilities can complete testing within time and budget? Describe how your solution adapts to the needs of team members who know scripting and those who don t? Before Scenarios No automation/performance tools and more tests than time to run Patchwork of tools that don t meet current business needs Current testing tools don t support new product initiatives (web/mobile/new code) Current tools do not support evolving 3 rd party releases (i.e., new version of IE, Chrome, 3 rd party controls) No automation/performance tools and more tests than time to run Patchwork of tools that don t meet current business needs Current testing tools don t support new product initiatives (web/ mobile/new code) Current tools do not support evolving 3 rd party releases (i.e., new version of IE, Chrome, 3 rd party controls) Tools difficult to use, steep learning curve for team members No automation/performance tools and more tests than time to run. Patchwork of tools that don t meet current business needs. Difficult to use, steep learning curve for team members Doesn t create robust test cases for people with limited skill sets Solution provides limited extensibility; it does not meet specific testing needs of experienced testers Negative Consequences Defects or performance issues released in production software, increasing support and development costs. Poor metrics and reporting. Lost pace in the market due to time consuming testing process, lagging development and missed release dates. Lost QA time spent maintaining tests based on new internal builds or third party releases. Defects or performance issues released in production software, increasing support and development costs. Poor metrics and reporting. Lost pace in the market due to time consuming testing process, lagging development and missed release dates. Lost QA time spent maintaining tests based on new internal builds or 3 rd party releases. Costly and difficult to hire, longer ramp time, lower productivity and greater cost of turnover. Defects or performance issues released in production software, increasing support and development costs. Poor metrics and reporting. Costly and difficult to hire, longer ramp time, lower productivity and greater cost of turnover. Poor release quality. Employee frustration due to inability to use tools effectively. 16

17 2015 SmartBear Confidential Test Discovery Guide Value Driver Improve Quality Across Mobile, Web and Desktop Increase Speed to Delivery Ensure Predictable Performance Improve Flexibility and Ease of Use Across All Skill Sets Discovery Questions (Future State) Describe what you need to get through tests in time. How do you envision your testing tools working together effectively? Explain what you need to effectively support new initiatives. Tell me what you would need to effectively support 3 rd party releases. Describe what you need to get through tests in time. How do you envision your testing tools working together effectively? Explain what you need to effectively support new initiatives. Tell me what you would need to effectively support 3 rd party releases. Describe what you need to get through tests in time. How do you envision your testing tools working together effectively? Describe what you need to get through tests in time. Explain what you need to ensure team members with different skills sets are able to test effectively. Tell me what you need to ensure the solution meets the requirements of both new and experienced testers. After Scenarios Tools enable test completion in the allocated time. Easily create, manage and run tests. Support for all technologies/platforms (desktop, mobile and web). Confidence that vendor stays current with market changes. Tools enable test completion in the allocated time. Easily create, manage and run tests. Support for all technologies/platforms (desktop, mobile and web). Confidence that vendor stays current with market changes. Tools enable test completion in the allocated time. Easily create, manage and run tests. Products support different skill sets and responsibilities Employees with different skill sets are able to create robust test cases Solution scales to different needs easily and quickly Positive Business Outcomes Reduced support and development costs. Efficient and predictable testing process speeds up development. Test maintenance is not time consuming or cumbersome. Reduced support and development costs. Efficient and predictable testing process speeds up development. Test maintenance is not time consuming or cumbersome. Reduced support and development costs. Efficient and predictable testing process speeds up development. Test maintenance is not time consuming or cumbersome. Efficient and predictable testing process speeds up development. Test maintenance is not time consuming or cumbersome. Quick ramp up time, higher productivity, decreased cost of turnover. Discovery Questions (What s Required to get there?) Describe what you need from a tool to complete tests on time. Explain the level of integration you would like among your tools. What platforms/technologies do you need to support? How do you judge if the vendor will stay current with market changes? Describe what you need from a tool to complete tests on time. Explain the level of integration you would like among your tools. What platforms/technologies do you need to support? How do you judge if the vendor will stay current with market changes? Describe what you need from a tool to complete tests on time. Explain the level of integration you would like among your tools. Tell me how you think products can support different skill-sets. Describe how you ensure if emp-loyees with limited coding skills can create robust tests. Describe how you know if a tool can be extended to provide specific functionality. 17

18 Test Discovery Guide 2015 SmartBear Confidential Value Driver Improve Quality Across Mobile, Web and Desktop Increase Speed to Delivery Ensure Predictable Performance Improve Flexibility and Ease of Use Across All Skill Sets Required Capabilities Automate tests across required environments and configurations. Well-integrated tools. Flexible tools that can adapt across new technologies/platforms. Timely support for evolving 3 rd party releases. Automate tests across required environments and configurations. Well-integrated tools. Flexible tools that can adapt across new technologies/platforms. Timely support for evolving 3 rd party releases. A central place to manage testing efforts. Automate tests across required environments and configurations. Well-integrated tools. A central place to manage testing efforts. A central place to manage testing efforts. Easily create custom extensions. Create robust tests without writing scripts. A central place to manage testing efforts. Metrics Hours spent testing per release. Delta between planned and actual testing release date (# of missed deadlines). Traceability matrix, test coverage. Technology/Platform support (checklist). Track record of vendor supporting new technology; Reusability of test cases. Time to support case resolution. # of defects found post deployment (by customer or internally). Delta between planned and actual testing release date (# of missed deadlines). Traceability matrix, test coverage. Technology/Platform support (checklist). Hours spent testing per release. Track record of vendor supporting new technology; Reusability of test cases. Time to support case resolution. Technology/Platform support (checklist). Hours spent testing per release. Track record of vendor supporting new technology; Reusability of test cases. Time to support case resolution. # of defects found post deployment (by customer or internally). Cost associated with rehire, ramp time for a tester. Delta between planned and actual testing release date (# of missed deadlines). Hours spent fixing defects, directly impacting revenue. # of missed testing deadlines due to long UAT cycles, which impacts revenue. How We Do It (Satisfy the Required Capabilities) Automate Tests (A Central Place to Manage Testing Efforts) Easy to create tests that can run on different platforms, environments, devices. Record and playback allows people to create automated tests without deep technical knowledge. Scripting, keyword driven test and extensive set of checkpoints for advanced users. Distributed testing. Well Integrated Tools QAComplete and TestComplete integration provides a single interface for managing, running and analyzing all kinds of automated and manual tests. Full Traceability across manual and automated Tests allows QA Managers to run traceability reports to ensure proper test coverage exists for requirements. Automate Tests (A Central Place to Manage Testing Efforts) Easy to create tests that can run on different platforms, environments, devices. Record and playback allows people to create automated tests without deep technical knowledge. Scripting, keyword driven test and extensive set of checkpoints for advanced users Distributed testing. Automate Tests (A Central Place to Manage Testing Efforts) Easy to create tests that can run on different platforms, environments, devices. Record and playback allows people to create automated tests without deep technical knowledge. Scripting, keyword driven test and extensive set of checkpoints for advanced users. Distributed testing. 18

19 2015 SmartBear Confidential Test Discovery Guide Value Driver Improve Quality Across Mobile, Web and Desktop Increase Speed to Delivery Ensure Predictable Performance Improve Flexibility and Ease of Use Across All Skill Sets Flexible Tools that Can Adapt Across New Technologies/Platforms Well Integrated Tools How We Do It (Satisfy the Required Capabilities) Test native, web and hybrid android applications and take advantage of enhanced functionality like Visualizer and Geolocation on IOS 8. Products have common UI rather than partnership. Timely support for Evolving 3rd Party Releases Rapid release schedules. QAComplete and TestComplete integration provides a single interface for managing, running and analyzing all kinds of automated and manual tests. Full traceability across manual and automated tests allows QA Managers to run traceability reports to ensure proper test coverage exists for requirements. How We Do It Better Automate Tests (A Central Place to Manage Testing Efforts) TestComplete includes images of the tested application captured during the test run and originally recorded screenshot; Particularly helpful in spotting the difference visually in case a test case fails. Cross browser testing. No need to re-record the test case in case of change. More scripting language support. Faster test execution. Distributed testing. Well Integrated Tools Common UI for testing web/desktop/mobile applications; integrated tools for scheduling, running, managing and reporting on all manual and automated test cases at a lower cost. Automate Tests (A Central Place to Manage Testing Efforts) TestComplete includes images of the tested application captured during the test run and originally recorded screenshot; Particularly helpful in spotting the difference visually in case a test case fails. Cross browser testing. No need to re-record the test case in case of change. More scripting language support. Faster test execution. Distributed testing. Automate Tests (A Central Place to Manage Testing Efforts) TestComplete includes images of the tested application captured during the test run and originally recorded screenshot; Particularly helpful in spotting the difference visually in case a test case fails. Cross browser testing. No need to re-record the test case in case of change. More scripting language support. Faster test execution. Distributed testing. Selenium integration. Flexible Tools that Can Adapt Across New Technologies/Platforms Superior object recognition. Access to device data (i.e., orientation, geolocation, Wi-Fi and locale). Consistent gestures. Timely Support for Evolving 3rd Party Releases Well Integrated Tools Common UI for testing web/desktop/mobile applications; integrated tools for scheduling, running, managing and reporting on all manual and automated test cases at a lower cost. Selenium integration. Management s commitment to release early. 19

20 Test Discovery Guide 2015 SmartBear Confidential Value Driver Improve Quality Across Mobile, Web and Desktop Increase Speed to Delivery Ensure Predictable Performance Improve Flexibility and Ease of Use Across All Skill Sets Industry Awards Industry Awards Industry Awards Customer References/Case Studies Proof Points 2014 Dr. Dobb s Award Received award for excellence in automated testing Customer References/Case Studies Numara Software Problem: Limited time, limited resources and issues with test coverage Solution: TestComplete Results: The automated scripts created using TestComplete can run in several hours instead of manually in a few days G3 Technologies Problem: Improve software quality and enhance productivity Solution: TestComplete, QAComplete Results: Seamless integration with TestComplete provided insight and control over manual and automated testing efforts. At the average American company, worker productivity improves maybe 2 to 3 percent per year. At G3T, we have to do even better than that so we can continue to thrive and grow profitably Dr. Dobb s Award Received award for excellence in automated testing Customer References/Case Studies Numara Software Problem: Limited time, limited resources and issues with test coverage Solution: Test- Complete Results: The automated scripts created using TestComplete can run in several hours instead of manually in a few days 2014 Dr. Dobb s Award Received award for excellence in automated testing Analyst Reports/Studies Gartner Magic Quadrant: SmartBear has a full life cycle view of software quality, including application performance management, as well as 3 rd party integration and a strong developer community program. SmartBear provides solid support for migrating assets from HP solutions into its products. It offers an attractive pricing model, with price points that address different markets. Customer References/Case Studies Numara Software Problem: Limited time, limited resources and issues with test coverage Solution: TestComplete Results: The automated scripts created using TestComplete can run in several hours instead of manually in a few days QualityMine Corporation Problem: Switching between different tools to automate tests Solution: TestComplete Results: A lot of tools limit users with the record and playback paradigm. For more serious test automation tasks, TestComplete provides the flexibility to implement automation frameworks that are highly reusable and maintainable across builds and different applications that can quickly boost automated test coverage and maintain existing automated test suites with ease. 20

21 2015 SmartBear Confidential Test Differentiator Test Differentiator Type (U,C,H) Differentiator Customer Value Defensibility Qualifications Unique Selenium Integration Add, edit, and run on Selenium WebDriver tests. Use screenshots to debug Selenium scripts easily. Unified execution and reporting of all testing activities. Collaboration between development and QA reduces time for test creation. Extended test coverage, traceability reporting. Dana Mohapl, Igloo Software Solution: Selenium WebDriver Integration with Test- Complete Results: The new TestComplete WebDriver Integration allows us to reuse and extend our scripts, while leveraging all the benefits of TestComplete. The built-in TestComplete reporting gives us more time to code on actual tests as we don t have to search for or develop reporting libraries. Finally, the screen shots provided by Test- Complete help us better understand why a Selenium WebDriver test failed. If you were able to leverage Selenium test cases built by developers, how much time would it save your testers? How do you incorporate results of tests run by developers into your test coverage reporting? Describe how you provide traceability across manual and Selenium automated tests. Minh Truong, DiCentral Solution: Selenium WebDriver Integration with TestComplete Results: Selenium support is a great feature in version I can integrate my TestComplete test scripts with Selenium scripts that I already developed in NUnit. This integration saves me a lot of time.» Comparative Single Platform for All Tested Applications Test mobile/web/ desktop applications through a common UI. Revenue Faster release cycles speed product delivery. Reduced costs due to fewer tools. Reduced training and ramp time. Modular purchasing, buy only what you need. Enables broader test coverage, higher quality products. Consistent process ensures on-time delivery. Alexie Karas, Independent Contractor for Kodak Solution: TestComplete s flexibility Results: What I get as a TestComplete user is: support for tests customization for different test environments, support for different versions and/or localized applications, crossbrowser support, support for direct data access and data-driven tests, seamless testing of the applications that are expected to run on different OS platforms (Windows, Android and ios) and communicate with one another. Describe how valuable it would be if you could test applications across multiple platforms from one UI. Describe what the impact on your release cycle would be if you had a single UI to test across web/desktop/mobile. Explain to me what challenges have prevented you from using an integrated tool set. Tell me about the challenges you have encountered using multiple products to manage your manual and automated test cases. 21

22 Test Differentiator 2015 SmartBear Confidential Type (U,C,H) Differentiator Customer Value Defensibility Qualifications Comparative Distributed Testing with TestExecute Run tests and report on them in a test lab, in virtual machines, or in the cloud. No need to buy platform, mobile, desktop, or web licenses. Revenue Faster release cycles speed product delivery. Broader testing coverage. Reduced support costs. Leverage cloud resources or virtual systems for testing. Broader testing coverage. Bengal Development Solution: TestComplete and TestExecute Results: No bottlenecks as customers don t have to wait for new test scripts to be built or be tested on a single test machine. If you could run the same test in parallel across multiple environments what would that mean for your testing schedule? Describe how you validate proper deployment of your application. Tell me about how you combine and run a variety of tests simultaneously. Customer validation testing. Comparative Mobile Gestures. Access to device data. Object-oriented tests. Faster instrumentation. Revenue Improved user experience drives adoption. Less time rebuilding image based tests that fail after changes are made. Reusable gestures ensure consistency of tests across mobile platforms. Access to device data allows for more robust test cases, better mirroring user experience. Nathan Ginoza, Costar Group Solution: easy ios instrumentation Results ios instrumentation with scripting in TC 10.5, works beautifully. Tested on ipad 3, ipad air, iphones, 4s, 5, 5s, 6 and 6 plus running ios 7.X and 8.X. This is the best news we can give to our development team who we can now fully support. This is great news that we can now handle it ourselves. Our developers now will only have to create a manual test build for QA, and then we will grab the IPA file and instrument it through scripting. How do you test the gestures of your mobile app? How do you test against landscape and portrait orientation against different devices? Describe the preparation you do before you start testing your mobile app. Explain what happens to your test cases when objects move in your applications. Object based testing reduces the risk of tests breaking due to changes introduced during development. Faster instrumentation reduces tool abandonment due to the time required to configure applications for testing. Comparative Rich Testing Capabilities Object-based recognition Checkpoints (large pre-configured list and custom options) Flexibility Revenue Faster regression testing cycles Reduced test maintenance and rework Test cases created quicker using predefined checkpoints Optiscan Solution: Test implementation was rarely blocked due to flexibility Results: Flexibility offered by TestComplete through plug-ins is perhaps its biggest advantage over its competitors. If ever during the implementation of a test I d stumbled across an area of functionality that is particularly hard to automate, I ve found a plug-in that has made the process much simpler. How do you prevent your tests from breaking when development changes the code or a feature (objects move, etc.)? If you had a list of pre-defined checkpoints describe how that would help you validate the expected state of your application. 22

23 2015 SmartBear Confidential Test Differentiator Type (U,C,H) Differentiator Customer Value Defensibility Qualifications Comparative More comprehensive test cases with flexibility to customize based upon unique application requirements Fast responses to questions means test implementation is rarely blocked. Comparative Test Visualizer Capture screenshots of actions duing recording and playback. Reduce the time of debugging by visually representing the differences between expected and actual state. Reproduce problems with visual audit trail of test steps. Interact with recorded images of test steps to quickly modify/edit them without scripting. Optiscan Solution: Test visualizer Results: Image comparison functionality in TestComplete meant ima-ges captured during Test procedures could be verified in review mode of the application. This made reviewing data entry and persistence easier. When a test fails, how much time would it save you in working with development if you could attach visual logs to your test results? Tell me about how you capture and track the steps leading up to a failure. How easily can you modify the step/flow of a test on the fly? Comparative Integrated Manual/Automated Testing Full traceability across manual and automated tests. Identify gaps in testing and prevent duplication of effort by managing all tests from a central location. Quickly assess testing progress across projects. Drive quality by connecting test results and defects to requirements (traceability). More effectively report on test coverage (e.g., what s covered by manual/automated, what s missing). Fidelity National Information Services Solution: Combine Manual Testing With Automation Results: Cut down maintenance costs drastically. Describe how you ensure that your test plans cover your requirements. How do you manage execution of manual and automated tests? Explain how you combine reporting across manual and automated tests. How do you ensure that your manual and automated tests are not overlapping on the same requirement? Holistic Timely Access to Technical Support and Rapid Resolution Customer Success team ensures satisfaction post purchase. Documentation. Reduced support costs. Decreased downtime. Dedicated internal resources, multiple support channels. Nathan Ginoza, Costar Group Solution: TestComp-lete Mobile 10.5 Results: The documentation for TestComplete 10.5 is very good and descriptive. It contains screen shots and sample scripts that guided us through the process. The development team did a really good job on this. ARUP Laboratories Solution: TestComp-lete Results: One pleasant surprise we have experienced with Test- Complete is the quality and timeliness of customer support. A big fear about support is that, once the product is purchased, support will become sporadic and low quality. TestComplete support has exceeded our expectations since we purchased the product. Describe what happens when something goes wrong with your tools. If something goes wrong, how would you expect your vendor to react? Tell me about the resources a vendor can provide to ensure your needs are met. 23

24 Test Differentiator 2015 SmartBear Confidential Competitor SmartBear Strengths and Weaknesses How They Compete Against Us Our Strategic Response HP-UFT + Easier to use for non-technical users Leverage their brand and history + Cost of implementation for deals larger than $50k + Integrated with Ready! API - Brand Recognition Migration from HP to TestComplete is expensive and labor intensive Largest market share, well known SmartBear has a rapid Product Feature Delivery model and we quickly introduce new market requirements in very short timeframes e.g. support for all latest browsers, new mobile capability etc. EggPlant TestPlant + TestComplete allows you to interact with the object layer of an application, which makes tests more robust - Reuse tests across interfaces, operating systems and devices Similar positioning as SmartBear s: Get to market faster, with higher quality, less effort, and clearer traceability They don t position their products on lower cost No Selenium Integration Image based testing; regression testing may fail for complex testing scenarios Integrated API functional testing solution Ranorex We are more well known Ranorex has a cheaper one cost option for all functionality Functionality for newer technology One License, all technologies Image or object recognition C# and VB.NET - Ease of use and maintenance of tests Modularized pricing allows you to customize package based upon technology needs Support for multiple scripting languages is more adaptable to different skill levels Customer Objection Our Response Proof Points It is too expensive to pay for an automation tool How much time are you spending during testing and how many defects are getting missed? Tell me about the cost of those defects being released. ARUP Laboratories Problem: Other automated testing applications cost thousands of dollars rather than a few hundred dollars per license, and the others do not offer a substantial improvement in features. Solution: TestComplete Affordable Solution Results: Better ROI We already have an automation tool How much time do your testers spend working on automation versus actually testing? Adobe Systems Problem: Customer owned one license for Silk but it was already in use and did not have the budget to buy more licenses Solution: TestComplete Results: Customer decides to try TestComplete and buys three licenses 24

25 2015 SmartBear Confidential API Readiness Discovery Guide API Readiness Tools Ready! API Discovery Guide Value Driver Enable Parallel Development and Testing Through Service Virtualization Expose Security Vulnerabilities in Services Improve End-to-End API Quality Ensure API Performs Predictably Under Load Discovery Questions (Current State) Describe your current approach for testing APIs. How does that affect your ability to deliver projects on time? Explain how you work with APIs that are still in development. How does that affect your ability to get software to market quickly? How many testing tools do you currently use in your development process? Describe your current approach to testing APIs. How does that affect your ability to deliver projects on time? Tell me about your approach to API security. What would be the impact of a security breach? How many testing tools do you currently use in your development process? Describe your current approach to testing APIs. How does that affect your ability to deliver projects on time? How many testing tools do you currently use in your development process? Describe your current approach to testing APIs. How does that affect your ability to deliver projects on time? Explain how you prepare for load impact on your API. What is the impact to your end users if your API doesn t perform well? How many testing tools do you currently use in your development process? No formal API testing strategy No formal API testing strategy No formal API testing strategy No formal API testing strategy Before Scenarios Serial dependencies and a parallel process (API is in development when testing needs to occur) Existing tools aren t integrated, don t scale or provide required functionality Unknown security profile of API Existing tools aren t integrated, don t scale or provide required functionality Existing tools aren t integrated, don t scale or provide required functionality Unknown API load threshold Existing tools aren t integrated, don t scale or provide required functionality High number of defects or support tickets due to poor or unknown API quality High number of defects or support tickets due to poor or unknown API quality High number of defects or support tickets due to poor or unknown API quality High number of defects or support tickets due to poor or unknown API quality Negative Consequences Decreased speed in delivering projects, missed deadlines Difficulty isolating that the API is causing an issue (MTTR) Failures caused by security issues (number of breaches, number of support tickets) Difficulty isolating that the API is causing an issue (MTTR) Difficulty isolating that the API is causing an issue (MTTR) Missed SLAs due to unreliable performance under load Difficulty isolating that the API is causing an issue (MTTR) Discovery Questions (Future State) How would you best leverage existing data and tests across your tools? Describe the benefit of having integrated tools. Tell me about how you would make sure your application testing can happen at the same time as your API development. What would the benefit be of having a shared environment? How would you best leverage existing data and tests across your tools? Describe the benefit of having integrated tools. Describe what would make you feel confident that your API is secure. How would you best leverage existing data and tests across your tools? Describe the benefit of having integrated tools. Explain how you ensure that your projects are completed on time. How would you best leverage existing data and tests across your tools? Describe the benefit of having integrated tools. What would make you feel confident that your API can handle predicted request volume? Describe the benefits of knowing benchmarks and threshold limits. 25

26 API Readiness Discovery Guide 2015 SmartBear Confidential Value Driver Enable Parallel Development and Testing Through Service Virtualization Expose Security Vulnerabilities in Services Improve End-to-End API Quality Ensure API Performs Predictably Under Load After Scenarios Tools provide all the required functionality in a more integrated and modern solution Shared environment where developers and testers can work independently Tools provide all the required functionality in a more integrated and modern solution Identified security profile and understanding of vulnerabilities if they exist Tools provide all the required functionality in a more integrated and modern solution. Well understood test plans and more accurate scoping. Tools provide all the required functionality in a more integrated and modern solution Known API performance benchmarks and confidence that APIs can support determined load thresholds Positive Business Outcomes More complete test coverage and ability to reuse tests and projects across tools (fewer defects, test coverage is higher) Lower operational costs and faster delivery to production (environment/api usage costs reduced or avoided, more product releases in same amount of time) More complete test coverage and ability to reuse tests and projects across tools (fewer defects, test coverage is higher) Minimized risk of security breaches More complete test coverage and ability to reuse tests and projects across tools (fewer defects, test coverage is higher) Projects delivered on time, deadlines achieved More complete test coverage and ability to reuse tests and projects across tools (fewer defects, test coverage is higher) Predictable customer experience (SLAs achieved) Discovery Questions (What s Required to Get There?) Describe what would help you plan and accelerate your testing. Explain how you can simulate API behavior without having access to the API itself. Tell me about how you ensure continuity across all functional groups. Describe what would help you plan and accelerate your testing. What would give you enough information about your API s security vulnerabilities? Describe what would help you plan and accelerate your testing. Tell me about how you ensure continuity across all functional groups. Describe what would help you plan and accelerate your testing. How do you ensure that your API met all SLAs at the peak times? Ability to visualize your API and understand its structure Ability to visualize your API and understand its structure Ability to visualize your API and understand its structure Ability to visualize your API and understand its structure Required Capabilities Ability to create virtual services with simulated environments and error conditions for testing and developing Ability to simulate security attacks and get security scan reports Single platform that provides a seamless, integrated experience for all users Ability to simulate load scenarios and receive understandable load test results Single platform that provides a seamless, integrated experience Test coverage reports Test coverage reports Test coverage reports Test coverage reports Metrics Operational costs (e.g., servers, data transfer, API usage, environment setup) and CapEx savings # of defects Security test coverage reports # of defects # of support tickets Cost of ownership Cost of rework # of defects SLAs achieved (5 9s) # of defects # of support tickets # of support tickets MTTR # of support tickets MTTR MTTR MTTR 26

27 2015 SmartBear Confidential API Readiness Discovery Guide Value Driver Enable Parallel Development and Testing Through Service Virtualization Expose Security Vulnerabilities in Services Improve End-to-End API Quality Ensure API Performs Predictably Under Load Our Approach We have an open approach to end-to-end API Readiness that is easy to use, integrated and accessible, based on the leading industry API testing tool, SoapUI How We Do It (Satisfy the Required Capabilities) Ability to Visualize API and Understand Its Structure Create test skeletons from service descriptions or discovery Build tests based on skeleton Generate coverage report Ability to Create Virtual Services with Simulated Environments and Error Conditions for Testing and Developing Create virtual services via Discovery, Description, Scratch Set up behavior patterns and error conditions to simulate server issues Share virtual services with other team members Ability to Visualize API and Understand Its Structure Create test skeletons from service descriptions or discovery Build tests based on skeleton Generate coverage report Ability to Simulate Security Attacks and Receive Security Scan Reports Run Security Test Generator to simulate different load conditions Review security scan reports Build targeted custom security scans from scratch or by extending existing scan Ability to Visualize API and Understand Its Structure Create test skeletons from service descriptions or discovery Build tests based on skeleton Generate coverage report A Single Platform that Provides a Seamless, Integrated Experience for All Users Shared data sources for data driven functionality Reuse tests and projects across multiple tools Shared plug-in framework for customizations across tools Ability to Visualize API and Understand Its Structure Create test skeletons from service descriptions or discovery Build tests based on skeleton Generate coverage report Ability to Simulate Load Scenarios and Receive Understandable Load Test Results Create a load test from an existing functional test Use different load models and strategies to simulate different load conditions Receive real-time load test results, reports and export data for more analysis A Single Platform that Provides a Seamless, Integrated Experience for All Users Shared data sources for data driven functionality Reuse tests and projects across multiple tools Shared plug-in framework for customizations across tools 27

28 API Readiness Discovery Guide 2015 SmartBear Confidential Value Driver Enable Parallel Development and Testing Through Service Virtualization Expose Security Vulnerabilities in Services Improve End-to-End API Quality Ensure API Performs Predictably Under Load How We Do It Better Ability to Visualize API and Understand Its Structure Agnostic, open approach to displaying API structure. Ability to Create Virtual Services with Simulated Environments and Error Conditions for Testing and Developing Affordable solution that makes it easy to create and manage virtual services. A Single Platform that Provides a Seamless, Integrated Experience for All Users End-to-end approach to API Readiness to ensure API quality at each stage of the lifecycle. Ability to Visualize API and Understand Its Structure Agnostic, open approach to displaying API structure. Ability to Simulate Security Attacks and Receive Security Scan Reports No need for security expertise because hacks are simulated. Built-in security scans and security test generator are unique in the market. Ability to Visualize API and Understand Its Structure Agnostic, open approach to displaying API structure. A Single Platform that Provides a Seamless, Integrated Experience for All Users End-to-end approach to API Readiness to ensure API quality at each stage of the lifecycle. Ability to Visualize API and Understand Its Structure Agnostic, open approach to displaying API structure. Ability to Simulate Load Scenarios and Receive Understandable Load Test Results Reuse functional tests with pre-built load strategies. Customer References/Case Studies Customer References/Case Studies Customer References/Case Studies Customer References/Case Studies Proof Points D&B Problem: Our internal team develops prototype applications (mocks) used as starting points for clients that then tailor the applications to the way they need the APIs to function in their environment. There are also situations where clients tell us an API is acting differently from what they expected, or they need the API to do something new. Solution: Service Virtualization (Mocking) Results: We customize SoapUI Pro to replicate the environment and build a prototype with the necessary adjustments. D&B Problem: Needed to demo API functionality for clients (visualize API) Solution: SoapUI Pro Results: We use SoapUI Pro on a daily basis to define design-level specs for each client and help them with the implementation. Much of this work involves demonstrating the specific functionality offered through the APIs. SoapUI Pro makes it possible to deliver demonstrations that are easy for clients to follow. Greg Jones, CTO, MachineShop Problem: As the Internet-of-Things market grows, MachineShop is using Ready! API to test a wide variety of conditions and parameters of these Operational Technology (OT) devices and their interaction with our next generation middleware platform. Testing these individually was a long and burdensome process. Solution: Ready! API Results: We have been able to automate this testing for the wide variety of data sources we support to ensure our customers the high level of security and reliability they expect from our service. Raymond James Problem: Using multiple tools for testing and monitoring Solution: API Monitoring Results: AlertSite for API Monitoring ensures the quality of our APIs, and we particularly benefit from the ability to share test scripts across each tool. This streamlines our testing processes, and when we start monitoring application performance, we can leverage the same scripts used to test the applications prior to going live. D&B Problem: Needed to demo API functionality for clients (visualize API) Solution: SoapUI Pro Results: We use SoapUI Pro on a daily basis to define design-level specs for each client and help them with the implementation. Much of this work involves demonstrating the specific functionality offered through the APIs. SoapUI Pro makes it possible to deliver demonstrations that are easy for clients to follow. Cisco Problem: Needed to verify that their cable television customers could provide reliable video delivery services (load testing) Solution: LoadUI Pro Results: We were already familiar with just how well SmartBear solutions work, having relied on SoapUI Pro for several years. We also considered other load testing tools but liked how easy it was to use LoadUI Pro and how it integrates seamlessly with SoapUI Pro. For load tests, we can adopt the same testing scripts used by our developers during the application development stage. Greg Jones, CTO, MachineShop Problem: As the Internet-of-Things market grows, MachineShop is using Ready! API to test a wide variety of conditions and parameters of these Operational Technology (OT) devices and their interaction with our next generation middleware platform. Testing these individually was a long and burdensome process. Solution: Ready! API Results: We have been able to automate this testing for the wide variety of data sources we support to ensure our customers the high level of security and reliability they expect from our service. 28

29 2015 SmartBear Confidential API Readiness Differentiator Ready! API Differentiator Type (U,C,H) Differentiator Customer Value Defensibility Qualifications Unique Single Platform for API Readiness Functional testing Load testing Security testing Service virtualization Revenue Deliver products to market faster, with fewer defects One solution/one expe-rience, minimizing ramp up time Subscription based licensing, minimal upfront cost Raymond James Solution: API Monitoring Results: AlertSite for API Monitoring ensures the quality of our APIs, and we particularly benefit from the ability to share test scripts across each tool. This streamlines our testing processes, and when we start monitoring application performance, we can leverage the same scripts used to test the applications prior to going live. How do you manage with multiple point solutions? Explain how you integrate the process across different functions. Describe how you ensure consistency, simplicity and accuracy across all the functions. Tell me about how you reuse tests and data across different point solutions. Reuse of assets means fewer chances for error Fewer vendor contracts/relationships Comparative Minimal Learning Curve Easy virtual creation Code completion Built-in security scans Editors Revenue Deliver products to market faster with fewer defects No need for professional services or training Shorter time to productivity Cisco Solution: Ease of use/minimal learning curve Results: In addition to wanting to use the same tool as many of our customers and partners, our application development and testing teams found SoapUI Pro is very easy to learn and work with. For example, our developers and testers can import any WSDL. This increases the adoption rate, which means we have another tool our teams use consistently. Tell me about your on-boarding process for new tools and for new hires. What is the ramp-up time? Describe the training required to get upto-speed with new tools. How easy is it to hire new staff with existing tool knowledge? How many of your testers are co-ders? Fewer human errors in developing and executing tests Less resistance to tool adoption 29

30 API Readiness Differentiator 2015 SmartBear Confidential Type (U,C,H) Differentiator Customer Value Defensibility Qualifications Data Driven Capabilities Functional testing Increases productivity Reduces overhead and rework Prime Therapeutics Solution: Data Driven Testing Results: SoapUI Pro is particularly stronger at enabling requests to read data from a data source file and then validating the response. QTP requires three weeks of development time using Visual Basic script code to achieve this, but SoapUI Pro reads data sources right out of the box. It took me only one hour to train two people. Explain how you test against large data sets and with complex data. How are you populating data into your tests? Comparative Service virtualization More coverage for various scenarios Liaison Technologies Solution: Data Driven Testing Results: When working with large-scale tests that include data intensive services and where data quality is critical, SoapUI Pro automatically gives us the data source to drive the tests. We just import the data source we want. We also benefit from the ability to read data through XML files. SoapUI Pro provides overviews of the actual data within XML messages as well as a simple interface for entering data into the requests. How are you validating your API calls against a back-end database? Describe how you simulate large data sets. How often do you recreate the same data sets for different tools? Comparative Point and Click Testing Functional testing Service virtualization Increases productivity Reduces overhead and rework More coverage for various scenarios D&B Solution: Service Virtualization (Mocking) Results: Our internal team develops prototype applications (mocks) used as starting points for clients that then tailor the applications to the way they need the APIs to function in their environment. There are also situations where clients tell us an API is acting differently from what they expected, or they need the API to do something new. We then customize SoapUI Pro to replicate the environment and build a prototype with the necessary adjustments. Describe how you string your API calls together. How many of your testers are coders? How do you pass a response from one operation to a request in another? Explain to me how much time you currently spend coding property transfers and assertions. Support Included in the pro version Increase productivity instead of looking for answers online Liaison Technologies Solution: Support Results: Because of how well SoapUI Pro works, we have not had to contact SmartBear all that often for technical support, but when we do, we always receive a timely response. Tell me about how you stay current with the latest tool updates. How do you maintain your installation? Holistic Upgrades included with subscription Auto-updater Development access to address bugs Explain how you address critical bugs. How do you triage product questions? What are your internal SLAs for response times? Describe how you on-board a new user. 30

31 2015 SmartBear Confidential API Readiness Differentiator Type (U,C,H) Differentiator Customer Value Defensibility Qualifications Holistic API Expertise Support and understanding of all protocols Open core platform based on SoapUI Most widely used API testing tool in the world Industry s most trusted API testing solution (your partners use it too). Gartner Report Magic Quadrant for Integrated Software Quality Suites 28 August 2014 Solution: API Testing Results: The tools are a good fit for technically adept organizations that want well-priced tools that weave naturally from development to operations. SmartBear is the driving force behind the broadly adopted open-source SoapUI, giving it a strong footing in the API testing market. Describe the importance of a large community of users (including open source) for tools that you use. How well does your current provider stay up to date with the current API technologies? Describe how your current provider integrates with other API tools. How mature is your current API testing tool? Competitor SmartBear Strengths and Weaknesses How They Compete Against Us Our Strategic Response SoapUI open source SOAtest by Parasoft + Data-driven testing capabilities, point/click, editors Free price point. + REST Discovery enables identification of undocumented APIs based on HTTP traffic + Any extensive functionality requires scripting or coding + On many issues community is unresponsive - Priced right Free - Wide user-adoption and stable community People get used to open source functionality. Base functionality may be enough for basic testing. + Security testing Parasoft is better at enterprise selling to + API discovery high-value customers. They also offer professional services to help these high-value + Plug-in framework for better customization customers onboard. + Comparable functionality at a lower cost and easier to use - Integration with other Parasoft code quality tools for run time error detection and code coverage Parasoft is better at partnering with other companies. Extensive functionality can be configured in Pro without coding.. Documentation only refers to Pro features. Support is provided to enable paying Pro customers. SoapUI NG Pro is a lot less expensive while providing all the same functionality and easier to use interface. With our free trials and freely available technical information end-users have an easier time setting up a proof-of-concept. SoapUI has a much larger user-base with a vibrant community in comparison with SOAtest. SoapUI NG Pro supports security testing, easy test creation from HTTP traffic, and a better customization framework. 31

32 API Readiness Differentiator 2015 SmartBear Confidential JMeter + JMeter requires advanced technical and product knowledge from its users Very popular open source tool and well known in software industry; has been Easier distributed load testing one-time install, drag and drop scenarios for + around since agents and reporting at the agent level + Out of the box reports easily customized to a host of analysis needs - - JMeter can be used for unit tests, load testing and production monitoring due to the flexibility of its scripting framework that supports fringe and complex technologies JMeter architecture is based on plug-ins. Most of its «out of the box» features are implemented with plug-ins. Off-site developers can extend JMeter with custom plug-ins. JMeter handles additional protocols that we do not yet support: FTP, LDAP, generic TCP connections and OS Native processes. SoapUI Pro and JMeter solve different needs: Jmeter load testing tool with a focus on web sites SoapUI Pro web service testing application for service-oriented architectures (SOA) and representational state transfers (REST) SoapUI Pro/LoadUIPro offers a very easy way to create tests for non-coders that can be used for load testing Customer Objection Our Response Proof Points It is too expensive to pay for an automation tool We can write an API testing tool ourselves The cost of the tool provides you with the ability to save time and add power to your tests using data driven testing as well as easy to use features like point-and-click. Automated testing requires an investment up front but it pays off over time as your staff has the time to do more complex testing. Homegrown tools need to be maintained and documented, which costs your company in terms of resources and overhead. Additionally, our tools have been in production for almost a decade so have incorporated industry and customer feedback over that time. The ability for any organization to build a tool with that level of functionality would take a long time. Prime Therapeutics Problem: Performance bottlenecks Solution: Ready!API Results: Reduced testing time and pinpointed performance bottlenecks through automation they went from a manual process of 6 hours to 20 seconds for the same number of tests. This allowed their testers and developers to spend more time debugging and improving API quality and business processes. Cisco Solution: Ready!API s easy-to-use interface and reporting engine Results: Fast adoption, which saved them the time it would take to build and maintain an internal solution. In addition, many of their customers are also SoapUI users so they were able to have synergy that they would not have had with a proprietary testing solution. 32

33 2015 SmartBear Confidential ServiceV Pro Discovery Guide ServiceV Pro Discovery Guide Target Buying Audience: QA/Development Manager Value Driver Enable Parallel Development and Testing Through API Virtualization Simulate API Behavior as Part of Application Testing Perform Application Load Testing without Impacting Production APIs Manage Costs and Availability Associated with 3 rd Party APIs Discovery Questions (Current State) For Consumer For Provider Describe how you ensure that your application doesn t fail when the underlying API fails. Describe how your application testers get their testing done while your API is still in development or is otherwise unavailable. Explain how you continue developing your API when another team needs to build and test against it. Tell me about how you load test your application without causing impact to the underlying APIs. Describe how you test and/ or load test your application when you rely on 3 rd party APIs that charge for use? Before Scenarios Application testers can t test the application completely until the API is built and tested by the API team. API development and testing is slowed by the need to provide access to external teams. Application testers don t test failure conditions with APIs. Load testing for the API has to be scheduled for off-peak times or it can t happen. Application testing incurs extra costs for 3 rd party APIs Negative Consequences Project delivery times are extended because activities have to happen sequentially API delivery time is extended, causing impact to the application delivery schedules. Bugs are left undetected until issues occur in production. Load testing is not done thoroughly/at all, causing issues in production Projects run over budget and are hard to estimate because of the unpredictability of testing Discovery Questions (Future State) Describe how you get access to an API that is currently under development. Explain how you give people access to your API without having them impact your API development. Describe how you know if your application will perform when the API responds with errors. Tell me about how you load test against an API that is unavailable. If you could simulate 3 rd party APIs, describe how that would impact testing. After Scenarios Testing and development schedules are not dependent on API availability (sandbox environment) Testing and development schedules are not dependent on API availability (sandbox environment) Application testing includes investigation of application behavior when the API fails. Load testing can be done in parallel with application testing and does not hit the actual API Extra costs are avoided by removing the need to send test transactions to the actual API 33

34 ServiceV Pro Discovery Guide 2015 SmartBear Confidential Value Driver Enable Parallel Development and Testing Through API Virtualization Simulate API Behavior as Part of Application Testing Perform Application Load Testing without Impacting Production APIs Manage Costs and Availability Associated with 3 rd Party APIs Positive Business Outcomes Reduced Costs (Projects Delivered on Time). Reduced Costs (Projects Delivered on Time). Greater resilience and a better customer experience due to more complete testing. Known application performance benchmarks and confidence that the application can scale. Projects delivered within budget. Discovery Questions (What s Required to Get There?) Describe how you would test against APIs that have been defined but are not fully developed. When an API is still in development and unavailable, describe what the testing team needs to start scoping and running tests. Walk me through the process of how you ensure your application is prepared for APIs that fail. Explain how you can predict your application will perform when the API is under peak load. If you wanted to simulate your test environment to avoid 3 rd party dependencies describe what you would need. Required Capabilities Ability to create virtual APIs from the service definition and share them in a sandbox environment with other development and test teams. Ability to create virtual APIs from the service definition and share them in a sandbox environment with other development and test teams. Ability to simulate various error conditions and environment settings in a virtual service. Ability to create virtual services from the API and run load tests using external data sources. Ability to record transactions run against 3 rd party APIs and create virtual services from those recordings that can be used as stand-ins for those APIs. Metrics Deadlines achieved. Deadlines achieved for both API and application teams. # of defects reported post-deployment. Benchmark established for application; SLAs achieved consistently. # of defects reported based on 3 rd party API dependencies. # of projects delivered within budget. How We Do It (Satisfy the Required Capabilities) Open Approach to End-to-End API Readiness Easy to use, integrated and accessible based on the industry leading testing tool, SoapUI. Quickly create a virtual service via API Discovery, an API Definition (in any format) or from scratch. Once the virtual service is built, drag and drop it to any Virtual Servers, letting as many teams access it as desired. Open Approach to End-to-End API Readiness Easy to use, integrated and accessible based on the industry leading testing tool, SoapUI. For web application testers, the ability to manipulate the API under the covers is often not available but this tool allows for easy creation of a simulation of the API to use for testing. Open Approach to End-to-End API Readiness Easy to use, integrated and accessible based on the industry leading testing tool, SoapUI. Manipulate settings that simulate overloaded APIs with ServiceV Pro by setting min/ max threads and network bandwidth in a simple interface. Open Approach to End-to-End API Readiness Easy to use, integrated and accessible based on the industry leading testing tool, SoapUI. Quickly create a virtual service via API Discovery, an API Definition (in any format) or from scratch. Once the virtual service is built, run load tests against the virtual API without worrying about impact to other applications that need the API. 34

35 2015 SmartBear Confidential ServiceV Pro Discovery Guide Value Driver Enable Parallel Development and Testing Through API Virtualization Simulate API Behavior as Part of Application Testing Perform Application Load Testing without Impacting Production APIs Manage Costs and Availability Associated with 3 rd Party APIs Our Approach We have an open approach that is affordable, easy and ready to use. Open Approach to Endto-End API Readiness Open Approach to End-to-End API Readiness Open Approach to End-to-End API Readiness Open Approach to End-to-End API Readiness Open Approach to End-to-End API Readiness How We Do It Better Compared to the other players in this space, solution is easy to use and affordable, letting teams be productive with the tool within minutes instead of weeks Compared to the other players in this space, this solution is easy to use and affordable, letting teams be productive with the tool within minutes instead of weeks. For development teams inside larger organizations, this service virtualization tool gives the lightweight solution needed to keep projects moving forward Simple user interface allows for the ability to choose error conditions and environment limitations (like low bandwidth or high volumes) to see if an application fails gracefully when the API fails underneath it. Compared to other vendors, this tool does not require special expertise to use and has minimal ramp-up time. Design focuses on being easy to use and quick to set up. Configuring a virtual service to mimic an overloaded API takes only minutes, allowing for the creation of scenarios that test an application s response to a slow or unresponsive API. Affordable Virtual Server tiers allow for easy set up of a virtual server intended just for load testing purposes. All of the performance testers and application testers can have access to the server and the virtual APIs residing there. Ability to deploy as many as desired and turn them on when needed Proof Points Customer Testimonials/Case Studies Robert Schneider, Partner at WiseClouds We are very excited about Ready! API, said Robert Schneider, Partner at WiseClouds, a leading training and advisory consulting services organization. We are offering both on-demand and live training as well as certification for Ready! API. Greg Jones, CTO, MachineShop Problem: As the Internetof-Things market grows, MachineShop is using Ready! API to test a wide variety of conditions and parameters of these Operational Technology (OT) devices and their interaction with our next generation middleware platform. Testing these individually was a long and burdensome process. Solution: Ready!API Results: We have been able to automate this testing for the wide variety of data sources we support to ensure our customers the high level of security and reliability they expect from our service. SmartBear Website, What is Service Virtualization? More and more companies are using service virtualization to improve productivity, reduce testing costs, and deploy higher-quality software in a shorter timeframe. In addition to emulating major software applications, 3 rd party services, and even whole backend systems, the virtual assets can also be reliably shared and used by your entire production team, facilitating more efficient parallel development practices. By quickly and easily removing dependency constraints across your organization through virtualization, you can gain a competitive advantage over other companies still waiting in the linear-development limbo. Customer Testimonials/Case Studies Robert Schneider, Partner at WiseClouds We are very excited about Ready! API, said Robert Schneider, Partner at WiseClouds, a leading training and advisory consulting services organization. We are offering both on-demand and live training as well as certification for Ready! API. Greg Jones, CTO, MachineShop Problem: As the Internet-of-Things market grows, MachineShop is using Ready! API to test a wide variety of conditions and parameters of these Operational Technology (OT) devices and their interaction with our next generation middleware platform. Testing these individually was a long and burdensome process. Solution: Ready!API Results: We have been able to automate this testing for the wide variety of data sources we support to ensure our customers the high level of security and reliability they expect from our service. 35

36 ServiceV Pro Differentiator 2015 SmartBear Confidential ServiceV Pro Differentiator Type (U,C,H) Differentiator Customer Value Defensibility Qualification Unique Seamless Integration with Functional API Testing Reuse of tests Can be built as foundation for tests Revenue Deliver products to market faster with fewer defects One solution/one experience minimizing ramp up time Subscription based licensing, minimal upfront costs Reuse of assets means fewer chances for errors Greg Jones, CTO, MachineShop Solution: Ready! API Results: As the Internet-of- Things market grows, MachineShop is using Ready! API to test a wide variety of conditions and parameters of these Operational Technology (OT) devices and their interaction with our next generation middleware platform. Testing these individually was a long and burdensome process. Using Ready! API, we have been able to automate this testing for the wide variety of data sources we support to ensure our customers the high level of security and reliability they expect from our service. Describe how you manage with multiple point solutions. Tell me about how you integrate the process across different functions. Explain to me how you ensure consistency, simplicity and accuracy across all the functions. How do you reuse tests and data across different point solutions? Fewer vendor contracts/relationships Comparative Minimal Learning Curve Easy virtual creation Clear easy-to-use server side simulations Drag and drop deployment Revenue Deliver products to market faster with fewer defects No need for professional services or training Shorter time to productivity Fewer human errors in developing and executing tests Press Release, Ole Lensmar, SmartBear We are expanding on the same model of affordable, yet extensible software that made SoapUI the leading API testing tool in the world. Ready! API provides a powerful set of individually purchased tools for API development, testing and monitoring in an integrated package. For areas such as service virtualization where applications have been priced out of reach for most developers, Ready! API provides a welcome relief and way to get the right tools into the hands of all developers. Describe your on-boarding process for new tools and for new hires. What is the ramp-up time? Tell me about the training required to get up-to-speed with new tools. How easy is it to hire new staff with existing tool knowledge? How many of your testers are coders? Less resistance to tool adoption 36

37 2015 SmartBear Confidential ServiceV Pro Differentiator Type (U,C,H) Differentiator Customer Value Defensibility Qualification Comparative Data Driven Capabilities Support for multiple data sources, including JDBC. Provides compatibility with application testing. Ability to create a variety of response scenarios by using different data sources. Increases productivity. Reduces overhead and rework. More coverage for various scenarios. SmartBear Software Website Get the same power from testing against virts that you get from functional testing directly against your APIs. With ServiceV Pro, you can identify external data sources for your virt to use in responses so your application testing can more accurately reflect real world examples. Tell me about how you test against large data sets and with complex data. Explain to me how you are populating data into your tests. How are you validating your API calls against a back-end database? Describe how you simulate your large data sets. How often do you recreate the same data sets for different tools? Holistic Support Included in the pro version. Upgrades included with subscription. Auto-updater. Increase productivity instead of looking for answers online. Most current version. Liaison Technologies Solution: Support Results: Because of how well SoapUI Pro works, we have not had to contact SmartBear all that often for technical support. But when we do, we always receive a timely response. Describe how you stay current with the latest tool updates. Explain to me how you maintain your installation. Tell me about how you address critical bugs. How do you triage product questions? Development access to address bugs. What are your internal SLAs for response times? How do you onboard a new user? Holistic API Expertise Support and understanding of all protocols. Open core platform based on SoapUI. Most widely used API testing tool in world. Industry s most trusted API testing solutions (your partners use it too). Gartner Report Magic Quadrant for Integrated Software Quality Suites 28 August 2014 The tools are a good fit for technically adept organizations that want well-priced tools that weave naturally from development to operations. SmartBear is the driving force behind the broadly adopted open-source SoapUI, giving it a strong footing in the API testing market. Describe the importance of a large community of users (including open source) for tools that you use. Explain to me how well your current provider stays up to date with the current API technologies. Tell me about how your current provider integrates with other API tools. How mature is your current API testing tool? 37

38 ServiceV Pro Differentiator 2015 SmartBear Confidential Competitor SmartBear Strengths and Weaknesses How They Compete Against Us Our Strategic Response CA (LISA) + Affordable Full virtualization of applications, APIs and the systems + Ease of use they rely on. - No system virtualization (lab environments) - Doesn t scale to enterprise level Ability to scale to large volumes of APIs and API transactions. Cost of product and implementation pays off in the lab; infrastructure savings. Lower price point and ease of use means you can spin up ServiceV and virtual APIs within minutes instead of weeks. Focus on APIs means we can bring our full expertise in this area to the solution and as part of our end-to-end API Readiness story. PARASOFT + Integration with API lifecycle products + - Plug-in framework for easy integration into other platforms No strong integrated development tools offering - No system virtualization (lab environments) Integration with API Testing and Development/Testing platform that allows for quality lifecycle. Full virtualization of applications, APIs and the systems they rely on. Lower price point and ease of use means you can spin up ServiceV and virtual APIs within minutes instead of weeks. Focus on APIs means we can bring our full expertise in this area to the solution and as part of our end-to-end API Readiness story. IBM GreenHat + Focus on API as core technology + Plug-in framework for easy integration into other platforms - Not integrated with non-api testing tools Continuous testing solutions as part of a DevOps story. Integrated with test automation and performance testing tools. Close integration with API design, testing and monitoring tools provides consistency across development and QA. Plugin framework allows for integration with multiple automated testing tools. Customer Objection Our Response Proof Points We already have another product. This isn t something we already do so we don t know who would be responsible for trying and buying. Our tool is complementary. As a simple, easy-to-use and affordable offering, you can easily install and use our product to create virtual APIs within your own environment without having to use the corporate Service Virtualization resources or licenses. Another benefit of ServiceV is its ability to consume SoapUI tests so you can use one tool for both API testing and API virtualization. The benefit of API virtualization is for both consumers and providers. Connecting with application development teams and API provider teams is essential in explaining the benefits of API virtualization. Mehdi Medjaoui, Co-Founder, OAuth.io and APIdays.io Conferences Problem: Disparate tools that aren t integrated with each other Solution: One consistent platform for all Smart- Bear API tools Results: The new SmartBear service virtualization module in particular makes enterprise-grade testing affordable for companies of all types and sizes. The beautiful user interface is exciting for enterprise software and the API Industry. IBM Developer Website Where many people fall short on their application testing is testing the application s response to API failures. If you are using 3 rd Party APIs and you only have access to their production APIs, it s very difficult to cause an API failure so you can t validate that your application fails gracefully when the API fails. Considering that the API is providing part of your functionality and therefore having a big impact on the user experience, this is an important gap in many development projects. One way to close that gap is to create virtual versions of the APIs you rely on. You can essentially create a sandbox where you can generate API failures, sub-optimal server conditions, and overloaded APIs without hitting any of the production APIs. 38

39 2015 SmartBear Confidential ServiceV Pro Differentiator We already use mocking. Mocking only simulates API request and responses. While it s beneficial in lightweight testing, it cannot provide you with the ability to simulate various conditions and API failures that will affect your application. SmartBear Website, What is Service Virtualization? Service virtualization is often confused with mocking and the creation of simulation stubs, but they re not the same thing. Mocks and stubs are fake software components that are used to imitate real software components for testing purposes. However, one of the biggest distinctions between mocking services and virtual services is that mocking functions tend to be context-specific, simulating a specific behavioral response to fulfill a certain development need at a certain time. Virtual services, on the other hand, can be deployed throughout the entire production cycle. Once they re created and exist as part of a project-wide test environment, virtual components eliminate the need for individual developers to write and rewrite their own mocks and stubs, saving time and effort for all concerned. SmartBear s Monitoring Tools AlertSite UXM Discovery Guide Target Buying Audience: IT Operations Manager/Business Owner Value Driver Easily Understand the Performance of All Web, Mobile Applications, APIs and Business Critical SaaS Applications Single Source of Metrics Correlating End User Experience and Root Cause Identification Identify Issues Affecting End User Experience Before Customers Easily Detect and Identify Problems Across All Tiers Discovery Questions (Current State) How do you monitor the experience your end users have with your web/mobile applications and APIs? Explain how you proactively measure critical user scenarios to ensure optimal user experience. Describe how you deploy and manage your (web, mobile) applications/apis today. Most customers tell us they use multiple APM solutions to help them monitor and manage network, infrastructure, web, third party and browser issues. How many solutions does your business rely on today? Walk me through the process you currently have in place to identify the root cause of a performance issue. What is an acceptable timeframe for you to identify the root cause of the problem? Describe the impact if you are not able to identify the root cause in the specified timeframe. Explain how you correlate specific errors or degradations that end users experience with root cause data from back end tiers. Explain how you identify performance problems today. What percentage of those problems are you able to detect before your customers do? When customers identify a problem, how do they let you know, and how do you deal with it? At times of low traffic, describe how you are still able to diagnose issues that would affect your end user s experience. When troubleshooting, what sources of data do you use? What are the typical problems you see affecting your applications (e.g., slowness related to network, ISP latencies, problems with application server components, database, third party content, data etc.)? Describe a time when performance was sub-optimal. What did it mean for the business and what information/tools did you use to identify the performance issue? 39

40 AlertSite UXM Discovery Guide 2015 SmartBear Confidential Value Driver Easily Understand the Performance of All Web, Mobile Applications, APIs and Business Critical SaaS Applications Single Source of Metrics Correlating End User Experience and Root Cause Identification Identify Issues Affecting End User Experience Before Customers Easily Detect and Identify Problems Across All Tiers Before Scenarios Spot monitoring only for peak hours/certain applications. Undetected, poor user experience or end users reporting errors. Unknown performance or availability of info for business critical SaaS applications. Inability to identify what data is accurate (finger pointing). Data sets are difficult to reconcile. Multiple UIs to learn. Customer complaints about web application errors/slowness. Drop in site conversions; dropped business transactions (e-commerce). Customer complaints on social media. Too much time spent trying to fight fires. Repeated war room scenarios when trying to identify issues. Inability to reliably and quickly identify the root cause of an issue. Negative Consequences Reduced revenue from user inability to complete business transactions. Increased costs due to missed SLAs and reduced revenue due to 3rd parties/cdns missing SLAs. Reduced productivity as a result of unavailability of applications. Increased MTTR. Missed projects. Increased resource costs. Increased operational costs (high call center volume) Decreased customer retention and new customer conversion Lost revenue due to incomplete transactions, fines, SLA violations Excessive downtime. Increased bottom line costs for additional resources. Increased time spent identifying performance issues causes missed releases and project timescales. Negative brand reputation Discovery Questions (Future State) Describe a reliable process for understanding and resolving performance issues (or when the end user experience is not desirable). Walk me through your ideal process for understanding API performance and root-cause at the same level as your web and mobile applications. In order to understand your user experience and gain visibility into the backend tiers, describe how you would like to work with your 3rd party outsourcing partner/ CDNs. How would you like to manage your business critical SaaS applications and measure them against SLAs? Describe how you would like IT, development and the business to work together to identify performance issues. What would be an appropriate way for the various stakeholders to understand and trust performance and health information of the various web, mobile applications and APIs? What would be an ideal way to understand if performance is impacting your end users? Walk me through how you would like to detect issues. Describe an ideal way for you to be able to detect errors and performance issues with your applications before your customers experience them. What would be an appropriate way for the various stakeholders to understand and trust performance and health information? How would you like to manage your 3rd party s content and data and measure them against SLAs? In order to understand your user experience and gain visibility into the back-end tiers, describe how you would like to work with your 3rd party outsourcing partner. 40

41 2015 SmartBear Confidential AlertSite UXM Discovery Guide Value Driver Easily Understand the Performance of All Web, Mobile Applications, APIs and Business Critical SaaS Applications Single Source of Metrics Correlating End User Experience and Root Cause Identification Identify Issues Affecting End User Experience Before Customers Easily Detect and Identify Problems Across All Tiers After Scenarios All web and mobile performance issues are detected before customers are affected. API scripts are monitored continuously alongside other applications. All critical apps and assets used by internal (company employees) and external users (customers, partners) are monitored. Confusion and blame is limited due to a single source of metrics. Reason and source behind conversion impact/changes is quickly identified. Consistent performance and defined process to detect and identify issues Performance issues are detected before customers are affected Ability to understand and analyze how 3rd party elements are performing and if they are SLA compliant Measuring application performance and understanding user impact enables greater control Increased team collaboration (internal, 3rd party, service providers) Positive Business Outcomes Business Owner. o Asset being monitored is green. Decreased MTTR. Decreased errors over time. Increased uptime Decreased errors over time Increased revenue Increased uptime IT/Ops. o Improved end user experience (within acceptable limits of business defined SLAs) and decreased errors or X% uptime. o APIs being monitored return with acceptable response times. Decreased resource costs. Increased productivity across IT/Ops/Dev teams. Decreased call center volume over time Increased conversion ratios Maintained brand reputation, increased customer retention Reduced call center costs Increased customer retention Reduced MTTR Discovery Questions (What s Required to Get There?) Tell me about your need to accommodate global locations for your business or customers. Describe how users and traffic flow over time with your applications. What 3rd party applications or APIs do your sites interface with? How long does it take to get to the root cause of a problem? How do you currently find out about issues with your applications? What performance metrics are you able to access for end user experience data and for back end tiers components? Describe how these metrics are tied together to provide a correlated view of your application environment. Describe your process for getting to the root cause of a problem. Describe how you measure the user experience. Explain how you can tell if your users are having performance issues. Describe any programs/projects that are in progress to reduce MTTR. Explain how you combine end user monitoring data with application tier component data to provide a complete picture of what is happening. Describe what happens when your operation team finds a problem. How do you currently correlate/ reconcile performance data/issues? 41

42 AlertSite UXM Discovery Guide 2015 SmartBear Confidential Value Driver Easily Understand the Performance of All Web, Mobile Applications, APIs and Business Critical SaaS Applications Single Source of Metrics Correlating End User Experience and Root Cause Identification Identify Issues Affecting End User Experience Before Customers Easily Detect and Identify Problems Across All Tiers Required Capabilities Proactively monitor/manage performance 24x7. Monitor across multiple geographies. Measure/track web, mobile, APIs, business critical SaaS applications. Monitor both outside and inside the firewall. Receive alerts when there is a problem. Have ability to chart and report on data over time. Monitor 3 rd Party Content/data/CDNs. Consolidated view of performance metrics across multiple sources. Correlated view of end user experience performance data with application component performance data. Ability to identify issues down to the level of a class, method, API call or DB SQL query on the application back end tiers. Performance data from the end user s point of view. Method to monitor transactions from a real browser. Ability to monitor transactions from different types of browsers. Ability to monitor different form factors for mobile enabled web applications (to accommodate different form factors). Ability to quickly identify root cause. Ability to link all end user and active performance data. Ability to distinguish 3rd party content or component sources and their performance. Metrics Down time associated with each application. # of Tools Used. Customer Retention/Conversion. Customer Retention. MTTR and Response Time. Response Time. Call Center Volume. MTTR. # of critical applications/assets monitored. Report Availability. Down Time, Response Time. How We Do It (Satisfy the Required Capabilities) 24x7 Across Multiple Geographies Multiple locations across the globe. Round the clock monitoring. Monitor All Web, Mobile Applications, APIs, SaaS Applications and Other Assets Using One Platform One view for all applications: web, mobile (native and mobile enabled web, hybrid, APIs, Business Critical SaaS apps), monitored using same platform. Inside and Outside Firewall Provide an appliance inside the firewall. Network for external monitoring. Alerting, Charting and Reporting Native alerting, charting and reporting part of platform. 3 rd Party Applications and Content Consolidated view Ability to drill down from end user view of interactions with applications, correlated to application server code level/transactional view. Unified platform that correlates data for all applications, across geographies and all browsers into single reporting/charting. End User Experience Metrics Provide performance metrics related to user experience. Provide real monitoring. Use real browsers for synthetic monitoring. Ability to support mobile enabled website monitoring to test for different mobile devices. Active Performance Data from an End User Point of View Synthetic monitoring using real browsers from various geographies. Root Cause Diagnostic capabilities with code level visibility of application server tier, related components and databases. Passive Monitoring Ability to carry out passive monitoring (gather all data for all users on the system) as well as, represent the back end tier data (in the same manner as the root cause capability). API monitoring and content views. 42

43 2015 SmartBear Confidential AlertSite UXM Discovery Guide Value Driver Easily Understand the Performance of All Web, Mobile Applications, APIs and Business Critical SaaS Applications Single Source of Metrics Correlating End User Experience and Root Cause Identification Identify Issues Affecting End User Experience Before Customers Easily Detect and Identify Problems Across All Tiers How We Do It Better Ability to monitor all critical applications/assets using one platform. Tools consolidation (end user experience and Root Cause). Value - robust functionality for price. No professional services needed to write scripts, deploy. Customer Success Manager. 24x7 Monitoring Across Multiple Geographies 85 nodes globally. Monitoring from independent nodes across geographies (not monitoring an application on AWS with our nodes on AWS). Inside and Outside firewall Measure from global network or appliance inside the firewall for internal applications and tie the data together. 3 rd Party Applications Reuse SoapUI scripts written for functional test directly as monitoring scripts; Monitor APIs just like the rest of the application. Use AlertSite s content views to understand how 3 rd party providers or CDNs are affecting your application. Consolidated View Active monitoring data correlated with Root Cause data provides end to end view. Support for.net, Java and PHP as application platforms. Root cause analysis for web, mobile and APIs. End User Experience Metrics DejaClick script recorder real in browser script builder to emulate user interactions. PUX metrics designed to provide data to determine user experience; above the fold, object level response times. Active performance data from an end user point of view In browser recorder records real user interactions to monitor with real browsers (DejaClick). Root Cause Diagnostic capabilities with code level visibility of app server tier, related components and databases. Passive Monitoring Provides actual user performance data, which is tied to the active data; Provides information about the impact of issues on actual users. 43

44 AlertSite UXM Discovery Guide 2015 SmartBear Confidential Value Driver Easily Understand the Performance of All Web, Mobile Applications, APIs and Business Critical SaaS Applications Single Source of Metrics Correlating End User Experience and Root Cause Identification Identify Issues Affecting End User Experience Before Customers Easily Detect and Identify Problems Across All Tiers Proof Points Analyst Reports Gartner Magic Quadrant SmartBear Software has been positioned as a Niche Player in the 2014 Magic Quadrant for Application Performance Monitoring. AlertSite UXM delivers a user-centric, unified solution focused on making it easy for an application team to ensure the end user experience. Customer References/Case Studies Marsh McLennan Problem: Receiving customer complaints, even though IT operations showed green lights in the NOC Solution: Monitor from the browser to see end user experience Results: Found problem with 3 rd party analytics tool. The JavaScript that was getting content was slowing performance for the whole site. Customer References/Case Studies Deloitte and Touche Problem: 19 separate solutions that addressed their application performance and collectively, there was a lot of noise and confusion. Solution: Single platform for continuous monitoring and root cause analysis Result: Provided a consolidated view of performance, saving them money in the process, and reduced mean time to resolution by 35% in the first year. UXM is now one of only 3 solutions used. Customer References/Case Studies Levi Strauss Problem: Expanding to new markets and needed customer experience to perform as well as in other areas Solution: Monitor from specific locations (UK and Germany) and evaluate CDNSs Results: Successful launch into new markets at expected performance levels. Customer References/Case Studies Party City Problem: Outages at peak times Solution: Implement synthetic performance monitoring from the browser 24/7, with alerting rules based on performance degradation Results: Immediate alerts to help discover any performance issues before peak times, after code changes. 44

45 2015 SmartBear Confidential AlertSite UXM Differentiator AlertSite UXM Differentiator Type (U,C,H) Differentiator (Technical) Differentiator (Business) Customer Value Defensibility Qualifications Comparative Fully Featured Active Monitoring Active (Synthetic) monitoring for a baseline, regardless of the traffic on the site. Global network of independent nodes (not hosted in AWS) for web, mobile and API monitoring (24x7). Multiple real browsers. Mobile device emulation and variety of carriers. In-browser script recorder no programming. Fully Featured Active Monitoring Monitor application performance from any location. Detect errors and bottlenecks before any impact on customers/ revenue. Monitor pre-deployment to detect issues. Manage 3rd party content and data (SLAs). Understand end user performance. Reduced down time because problems are identified and fixed faster. No need to build/maintain monitoring network. Detect problems before customers do. Fully configured and maintained network (customer does not need to maintain it). Party City Solution: Browser based monitoring Results: Fewer outages. Levi s l Solution: Global network l Results: Successful expansion into global market. How do you currently monitor your web performance? How is it providing all the coverage you need? How do you currently find out about problems? Describe the business impact when customers find a problem before you do. Load time metrics for individual objects above the fold. One view into state of all applications/assets being monitored. Up/down availability. See everything that supports 3 rd party (content, data). Receive alerts via assorted modalities. One platform for all types of applications. 45

46 AlertSite UXM Differentiator 2015 SmartBear Confidential Type (U,C,H) Differentiator (Technical) Differentiator (Business) Customer Value Defensibility Qualifications Unique Active Monitoring Combined with Root Cause Analysis Single interface for multiple sources of data Trace and correlate an end-user transaction from browser to server tier (end-to-end) Automatically discover the physical infrastructure and software components and display it Transaction tracing without stopping and re-starting back end servers Identify precise components and the module/line of code causing an error or possible degradation Active Monitoring with Root Cause Analysis Against Active/(Synthetic) Competitors: Single pane of glass that allows correlation of end user data with root cause data Passive Monitoring: Benchmark expected behavior and identify root causes of delta between benchmark and actual performance Against Passive Monitoring Competitors: Purpose-built, integrated solution Revenue Increased customer retention Increased productivity Reduced MTTR (Single platform for functional groups to share and easy data correlation and comprehension) Disney Solution: Previously used AlertSite alerts to identify an issue, before drilling into the problem, using legacy infrastructure tools. Transaction Tracing allows them to understand the cause of the performance problem within the initial alert. Results: This reduced MTTR and allowed them to understand performance issues before their customers found them Describe the process you go through to find the root cause of an application performance issue. How do you ensure that your APIs and third parties are performing as expected? Easily record end-user transactions, making the operations teams self-sufficient and reducing reliance on vendors or internal engineers Comparative Mobile Monitoring and Root Cause Identification Monitor mobile enabled websites (responsive design) Monitor native mobile applications through their APIs Monitor hybrid mobile applications using a combination of the above Monitor real devices in production Combine mobile, web and API data into one platform Mobile Monitoring and Root Cause Identification Detection of errors and degradation of end user experience Monitor application from different geographies and using native device types/carriers Correlated view of end user experience degradation with root cause identification Revenue Increased customer retention Reduced MTTR No blind spots Lowes Solution: Using AlertSite UXM and Perfecto Mobile Results: Able to diagnose/ understand issues on their back end tiers, reduced MTTR by quickly identifying bottlenecks and increased customer satisfaction A major retailer Solution: AlertSite UXM Results: Had a set of scripts monitor mobile experience. Solved the 3am problem and alerted operations of errors/ degraded performance. Root cause identification showed app server had issues with their MySQL backend database. Development resolved issue, reducing MTTR. Tell me about your mobile initiatives. Do you currently understand the performance of your mobile applications/ websites? 46

47 2015 SmartBear Confidential AlertSite UXM Differentiator Type (U,C,H) Differentiator (Technical) Differentiator (Business) Customer Value Defensibility Qualifications Comparative Behind the Firewall Monitoring with Same Features as Public SaaS Deployment (Insite) Monitor internal applications that cannot be exposed Monitor in pre-production Same level of functional granularity for APIs and application Monitor in a Secure Environment Increase productivity Find problems pre-deployment Ensure that internal SLAs are met and productivity is not effected Revenue Fix performance issues in preproduction Same system used in pre-production Raymond James Solution: Insite Results: 10 installations supporting secure transactions Edwards Credit Union Solution: On prem. monitoring appliance Results: Ability to monitor secure network from end user perspective, MTTR decreased 30% Tell me about the risks of poor performance on internal (employee-used) applications? Tell me how you monitor your business applications in a secure manner. Identify issues before deployment Comparative Better/Faster Scripting (Non Programmer) In-Browser Transaction Recorder record your actions to create and maintain scripts used to monitor Multi Browser No Programming Sophisticated scripts accommodated Editable Free use/try standalone Handles complex websites Easy to Use, Point and Click Recorder to Set Up Monitors Record and relax Ability to set up your own monitor Ability to monitor competitors yourself Less time spent to create scripts and be up and running Non-technical staff can create scripts, decreased resource costs Decreased dependency on skilled staff for implementation and maintenance Edwards Federal Credit Union Solution: DejaClick Results: Able to create scripts, including highly secure real time password protections BCBS TN Solution: DejaClick Results: Cut time spent maintaining scripts by 80% Westjet Solution: DejaClick Results: Able to bring scripting in house for better customer service, reduced call volume, increased productivity How much time do you spend now to get monitors up and running on the transactions you want to monitor? (Non-programmer) Describe the benefits you have received from being able to script quickly. Tell me about your process to monitor a different part of your site or a different competitor. Unique Connection to SoapUI for API Monitoring and Root Cause Analysis (test scripts) Complete Information to Manage APIs Crucial to Business Applications Re-use of test scripts, less labor Raymond James Solution: SoapUI integration Results: 800 APIs to monitor increased capacity and monitoring capabilities. Describe how you currently test and monitor web services. Use Soap UI test scripts to monitor production APIs Code level visibility for root cause identification Identify root cause of performance problems in APIs functional insertions Ensure performance of distributed applications Single solution Reduced MTTR Risk Single vendor Better coverage (using the same script) Easy Debit Solution SoapUI integration Results: Reduced MTTR by 62% Intrix Solution: SoapUI integration Results: Greater visibility into their API services Are you currently using SoapUI for testing your APIs? If you have a failure with your application can you determine if the error is in the API? 47

48 AlertSite UXM Differentiator 2015 SmartBear Confidential Competitor SmartBear Strengths and Weaknesses How They Compete Against Us Our Strategic Response Compuware/ Gomez/ Dynatrace + Single solution for active monitoring with root cause (and passive) + Faster time to value/easier to use and deploy (no professional services) SmartBear has cheap up and down monitoring only Web performance only (no APSs or APS) SmartBear can t scale appropriately Cost effective technology performance partner (trusted by organizations such as Disney, Raymond James, Jet Blue). End user focused performance management. - Deeper diagnostics capability - Bigger market presence, bigger field sales (safer choice) Gain detailed understanding of performance issues on web/mobile applications and APIs. Keynote + Single solution, faster Cheap up and down monitoring only + Control over the monitoring solution - Mobile solution (device anywhere), much bigger mobile presence - Industry standard benchmarking published Scalability concerns Kite is better scripting engine than ours No Mobile native application monitoring (testing) New Relic + Single solution No need for sophisticated synthetic up and down + No agent needed to get started - Huge market presence and spend (agents deployed) ecosystem - Robust real monitoring solution and easy deployment Synthetic is expensive Catchpoint + API Monitoring Expensive Many Keynote customers switched in large part due to DejaClick transaction recorder. They specialize in mobile but not APIs. Strategic partnership with Perfecto Mobile. Availability monitoring does not provide visibility for root cause identification. Business owner not capable of using New Relic. Find the root cause of a problem quickly, saving costs. They are a small company can they scale? + Browser Based Transaction Recorder + Root Cause Analysis - No Streaming - Network Performance Overkill Alertsite lost focus with all SmartBear portfolio They don t know retail as well as us Alertsite back bone only No scripting recorder. We have lots of e-commerce companies like Disney, etc. (Reference Proof Points).Geography is not as broad as ours. 48

49 2015 SmartBear Confidential AlertSite UXM Differentiator Our Response Proof Points Customer Objection I already have everything I need. Reference PBOs and use cases demonstrating how we have worked alongside other vendors and provided value to the customer by increasing revenue, decreasing costs, or mitigating risk. Who is the business owner? Do they feel that way? How many solutions are you using? Deloitte and Touche Problem: 11 solutions Solution: continuous monitoring with alerts and root cause Results: Reduced MTTR I already have an active monitoring system (Keynote, Gomez, etc.). Reference the PBOs and differentiators and reframe the discussion toward what the customer needs. West Jet Problem: Scripting is difficult, time consuming and done out of house, which is expensive Solution: In browser transaction recorder Results: Reduced time, costs and resources needed This is not a priority and/or I don t need it. Is this application or system mission critical? What is the impact of poor performance or failure with this application/system? Looking at your performance compared to your competition, you seem to be performing slower than X and Y. There are some key reasons why. Who can we work with to help them appreciate how customer experience affects your business? Who in the company cares about the user experience for this system/application? Disney Interactive Problem: Moved away from Alertsite to save money, using ping testing tools. Twelve months later they came back to SmartBear after a performance problem created outages Solution: Alertsite UXM Results: Increased their use of AlertSite UXM and using performance metrics. 49

50 SmartBear s Channel Management Team 2015 SmartBear Confidential SECTION IV: SMARTBEAR S CHANNEL MANAGEMENT TEAM EMEA Channel Director John McArdle E: [email protected] T: M: SmartBear Ireland Ltd, 3rd floor Dockgate, Unit 19, Merchants Road Galway, Ireland Partner Marketing Manager Christina Lowe E: [email protected] T: M: SmartBear Software Inc., Assembly Row, 450 Artisan Way, Somerville, MA 02145, USA Name Title Regional Scope Office Location Debbie Decoulos Channel Sales Manager Global Somerville, MA Tina McIntyre Channel Sales Manager Global Somerville, MA Michael Samuelian Director of Strategic Alliances Global Somerville, MA Suzanne Cella Channel Sales Operations Global Somerville, MA Details on the two Partner Addresses [email protected]: created to manage order correspondences and processing orders all orders are to be sent to this address [email protected]: general inquires, account management, deal registration, product/support questions, etc. 50

51 European Headquarters Global Headquarters Development Centres SmartBear Ireland Ltd 3rd floor Dockgate Unit 19 Merchants Road Galway Ireland Tel: Web: SmartBear Software, Inc. Assembly Row 450 Artisan Way 4th floor Somerville, MA Tel: Web: SmartBear Software, Inc. Hornsbruksgatan, 28 SE Stockholm, Sweden Tel: +46 (0) Development Centres Development Centres Development Centres SmartBear Software, Inc Johnson Road, Suite 6 Coconut Creek, FL US Tel: x.237 Fax: SmartBear Software, Inc N. Capital of Texas Hwy Suite I-310 Austin, TX SmartBear Software, Inc. Chernikov lane., d. 6B Tula, Russia Tel: +7 (4872) Fax: +7 (4872) [email protected]