Rational Quality Manager Evaluation Guide. Created: 29 th December 2010 This guide is based on: RQM

Transcription

1 Created: 29 th December 2010 This guide is based on: RQM

2 Contents Introduction...2 Installation...2 Setup...2 Option Option Typical Evaluation Criteria...3 Requirements Management...3 Test Management...3 Defect Management...4 Lifecycle Integrations...5 Non Functional...5 Additional Evaluation Criteria...6 Planning the Evaluation...7 Evaluation Milestones...8 Rational Quality Manager Overview...8 Rational Quality Manager Capabilities...9 Dashboards...9 Requirements...10 Test Planning...11 Test Construction...12 Test Implementation...13 Keywords...14 Test Environments...15 Test Execution...16 Test Lab Management...17 Defect Tracking...18 Test Reporting...19 Customisation...20 Next Steps...20 Additional Resources...21 Page 1

3 Introduction The purpose of this guide is to highlight key features and functionality of Rational Quality Manager and to enable quality professionals to successfully explore and evaluate it's functionality. Installation Rational Quality Manager is available for download from both the IBM and Jazz.net websites and can be evaluated for 60 days. Installation documentation is available on both sites but installing Rational Quality Manager is a simple affair and can be initiated by extracting the files, running the setup wizard and accepting the defaults Alternatively, your IBM representative may have supplied an automated installation including both sample and evaluation projects, these contain examples and guidance relating to the evaluation. In order to install this version, simply run the supplied batch file. The Jazz website can be found at and contains vast quantities of information on both Rational Quality Manager and the Jazz initiative including wikis, forums and a window into the development organisation, allowing customers unprecedented access directly to product management and the development teams. Setup Once Rational Quality Manager is installed, simply start the Rational Quality Manager server if required, open IBM Rational Quality Manager and login. Option 1: For those customers who downloaded the directly from either the IBM or Jazz websites, the default user ID and password are both ADMIN (notice the uppercase). On initial login, the ADMIN user will be redirected to the Rational Quality Manager set up screen, for evaluations, selecting the Fast Pass Setup is recommended. Accept all the default settings. As part of the setup process, a new user will be created and the ADMIN user disabled. The setup process will also create a sample project area named 'Quality Manager'. Option 2: For customers using the pre-packaged automated installation, the username and password are both rqmeval (notice the lowercase). Page 2

4 Typical Evaluation Criteria When evaluating a test management solution, there are a number of questions that are typically asked. The following evaluation criteria has been consolidated from questions posed to IBM from our customers. These questions have been taken from numerous RFI's and sanitised for use in this document. This shouldn't be considered a definitive list, the purpose of these sample questions isn't to ask only those questions that Rational Quality Manager is known to address, but to provide an unbiased view of the types of things that should be considered when choosing a tool. Additional space has been included in these tables as every customer will have individual needs specific to their organisation. Requirements Management ID Criteria Findings RM1 The system shall provide the ability to define requirements identified by a unique reference RM2 Requirements can be weighted based on risk and probability RM3 Requirements can be associated to test specifications (many to many) RM4 Requirement coverage can be monitored and reported against RM5 The system shall be able to import structured text from Microsoft Excel and Word file formats RM6 RM7 RM8 RM9 RM10 Test Management ID Criteria Findings TM1 Ability to define test specifications for a project and sub-project TM2 Ability to define test scripts for a project and sub-project TM3 It should be possible to import test assets from MS Excel TM4 Ability to define the environment the tests will be executed Page 3

5 TM5 TM6 TM7 TM8 TM9 TM10 TM11 TM12 TM13 TM14 TM15 Test scripts can be executed from any workstation within the chosen environment Ability to identify the sequence in which tests are to be run Ability to link screen shots and/or external files as evidence Ability to view the status for each step executed It should be possible to view a history of previous test runs Ability to link another document to a script (for example, the test data used in the script) test templates can be created for common test steps (e.g. logging on / off), these can be referenced from new test scripts The tool should be able to run Manual and Automated test scripts Defect Management ID Criteria Findings DM1 Ability to create a defect identified by a unique reference DM2 Ability to define which defect attributes are mandatory according to status DM3 Ability to add, edit and remove fields including dropdown list values DM4 Ability to customise the status/action workflow of the defect DM5 The tool should be able to easily query defects based on any attribute DM6 Ability to attach evidence to the defect such as screen shots and/or external files DM7 DM8 DM9 Ability to report on the defect status Ability to report defect status over time Ability to raise new defects at the point of failure DM10 Ability to configure notification DM11 Page 4

6 DM12 DM13 DM14 DM15 Lifecycle Integrations ID Criteria Findings IN1 Capabilities to support integration with requirements management tools IN2 Capabilities to support integration with test automation tools IN3 Capabilities to support integration with defect management tools A well-defined interface shall be provided for use IN4 by external systems. This shall be in the form of a comprehensive Application Programming Interface (API) IN5 IN6 IN7 IN8 IN9 IN10 Non Functional ID Criteria Findings NF1 Tool access is controlled by username and password NF2 Users permissions are determine by role (e.g. what they can do within the tool) It must be possible to create multiple projects NF3 within the tool, with each project requiring users to be added to the permissions group NF4 Ability to integrate with LDAP for user authentication NF5 Can be accessed from multiple locations by multiple users concurrently Page 5

7 NF6 NF7 NF8 NF9 NF10 Training courses for users should be readily available, either in-house or off-site Support facility providing telephone assistance as part of the annual support agreement The support service must include defined escalation procedures and contacts for such a procedure Additional Evaluation Criteria The following section can be used to capture any additional evaluation criteria specific to each individual customers requirements that wasn't captured in the previous sections. ID Criteria Findings AD1 AD2 AD3 AD4 AD5 AD6 AD7 AD8 AD9 AD10 AD11 AD12 AD13 AD14 AD15 Page 6

8 Planning the Evaluation The time taken to fully explore and evaluate a tool against a set of agreed criteria will vary as it will typically run parallel with existing work commitments. For that reason dates and durations have been omitted from the following plan. Its purpose is purely to document evaluation activities and includes optional milestones where your IBM representative can visit to address any questions arising from the evaluation. Page 7

9 Evaluation Milestones As was mentioned in the planning section above, your IBM representative will be more than happy to visit throughout the evaluation process and address any questions or concerns that have arisen as well as demonstrating and/or discussing some of the more advanced capabilities offered by Rational Quality Manager. Any milestone dates can be agreed with your IBM representative and captured below Milestone Evaluation set-up and Overview Date and Time Today's date Optional Milestone... Evaluation Progress Review Optional Milestone... Evaluation wrap-up and next steps Rational Quality Manager Overview Rational Quality Manager is a web-based centralized test management environment that allows quality professionals to effectively collaborate on all aspects of testing including test planning, construction, implementation, lab management and execution. Rational Quality Manager includes the ability to create a meaningful test plan directly within the tool based on an agreed set of requirements. Users can create these requirements directly in Rational Quality Manager, or requirements can be tracked through a dynamic link to requirements management tools such as IBM Rational RequisitePro or IBM Rational DOORs. Page 8

10 Test Plans can be customised to conform to a customers test processes but would normally include the requirements that are in scope and the test cases that will verify their functionality. Rational Quality Manager includes a rich manual testing capability which provides quality professionals with a rich text editor that can capture textual test steps alongside embedded images. Using this editor, testers can record the test steps that must be executed in order to validate a given function of their application. These test scripts can then be associated with test cases and executed. For those looking to automate testing, Rational Quality Manager can also integrate with test automation tools such as Rational Functional Tester, Rational Services Tester and Rational Performance Tester. To enable test reuse, Rational Quality Manager uses keywords, which allow testers to share common steps between tests. These can be manual steps or automated test scripts that are powered, for example, by IBM Rational Functional Tester (RFT) software. Creating hybrid manual tests with automated keywords is often a very simple way for manually focused test teams to leverage the power of test automation, without making a full commitment to a test automation framework. Rational Quality Manager provides each user with an individual dashboard that is continually updated, allowing each user to view the most current information that pertains to his or her work. In addition to this, a wide variety of formal and customizable reports are also available that can capture live and historic information at varying levels of granularity to help ensure that all team members have access to the information they need. Rational Quality Manager Capabilities The remainder of this is designed to support the evaluation activities defined above and will highlight key areas of Rational Quality Manager, enabling quality professionals to explore and evaluate it's functionality Dashboards Rational Quality Manager offers comprehensive dashboard capabilities, allowing users to see in real time, information that is relevant to the role they play in the software development lifecycle. For a Tester this may only extend to their task list and the status of any issues they have raised. Whereas for the Test Manager, it may include requirement and test coverage, team loading or reporting. Additional viewlets and tabs can be added to the users personal dashboard by clicking the and buttons respectively. Page 9

11 Once the button is clicked, the user is presented with a choice of viewlet types that can be added to their dashboard. Viewlet types include the users task list, project and team activities, requirements and test coverage, work item statistics as well as news feeds (ie. RSS) for monitoring user forums, blogs as well as commercial news sites. Each viewlet can the be individually configured to show different subsets of information such as requirements coverage by test plan. Viewlets can also be duplicated and moved between tabs, for example if a user wanted to view defects by priority and severity. It is also worth noting that most viewlets are interactive allowing users to drill though to the data used in those charts Requirements Rational Quality Manager allows quality professionals to capture those requirements relevant to testing in order to build traceability links to both test plans and cases, this ensures requirements coverage and allows for impact analysis should one of those requirements change. Page 10

12 As has already been mentioned in the overview, Rational Quality Manager integrates with both the industry leading requirements management tools, IBM Rational RequisitePro and IBM Rational DOORs (formally Telelogic DOORs). This allows quality professionals visibility of the requirements as they are synchronised between the external tools and Rational Quality Managers internal ones. In addition to this, requirements can also be entered directly into Rational Quality Manager allowing quality professionals to capture their testing requirements and test conditions and trace these to test plans and test cases. Rational Quality Manager provides a comprehensive query engine that allows users to query work item data such as requirements and defects in order to get the necessary data in the required format. Where appropriate, these queries can also be used to form the basis of a number of viewlet types that can be added to the users dashboard. Test Planning Rational Quality Manager includes the ability to create a meaningful test plan directly within the tool and removes the need for externally managed documentation. While traditional test plans are text based documents containing headings representing the information that needs to be captured, the test plans in Rational Quality Manager are accessible online enabling collaboration and sharing of information across the whole project team. These plans still include sections that should be completed prior to the plan being reviewed and approved, however, sections of the plan can be assigned to an individual in order to distributed the work load evenly across the test team. These tasks will then appear in the individuals task list awaiting completion. Because the Rational Quality Manager test plan is designed to be an alternative to the test plan document, Rational Quality Manager offers full rich text editor capabilities allowing the user include tables, images, various font sizes, styles and colours. Page 11

13 Each test plan can be based on a user-defined template allowing users to pick and choose the sections to be completed based on the size, complexity and formality of the project, ranging from a lightweight agile approach to a full traditional waterfall version. Regardless on the approach taken, Rational Quality Manager will provide complete traceability across the software development lifecycle ensuring that all requirements are fully validated. A typical test plan would normally include the requirements in scope for current phase of testing as well as document the types of testing and test cases that will be performed in order to validate the functionality. The approach to test planning may vary depending on the software development methodology being used. For example an iterative approach may result in a test plan per iteration where each iteration delivers a subset of the overall systems requirements. Test Construction Test Cases within Rational Quality Manager are reusable assets that can be included in multiple test plans throughout a software development project. The primary function of a test case is to document the test that is to be performed and capture the appropriate level of detail in order to allow any member of the quality assurance team to be able to perform it. Typically, this detail will consist of: Who : The person responsible for creating and maintaining the test. What : The test to be performed When : The iteration or phase in which the test should be performed Where : The environment in which the test is to be executed Why : The reason why the test is being performed Page 12

14 Rational Quality Managers test cases and plans include the ability to create user-defined categories in order to tag these assets with meaningful identifiers such as product, version, function and test phase. Test cases also include the concept of weighting. The weight is used to represent the complexity of the test. The level on granularity may vary (1-10, 1-100, etc) but the purpose is to differentiate between test cases in order to produce more accurate progress reporting. For example, registering a user on a website is a simple scenario and may take less than 30 minutes to create the test case whereas the scenario of applying for a mortgage may take a number of hours. A report showing completed test cases may indicate 50% of the work is complete when in reality, the easy test case (ie. 20%) is complete and more time consuming test case (ie. 80%) is still outstanding. Similar to test plans, Rational Quality Manager test cases include sections that should be completed prior to the test being reviewed and approved. While sections of the test case can be assigned to an individual, typically the entire test case would be given to a member of the quality team to complete. The test case will then appear on the individuals task list similar to those already mentioned. The quality team member can then mark this complete once the test case is ready. Test cases can also be based on a user-defined template allowing a user to pick and choose the sections to be completed based on the size, complexity and formality of the project. Test Implementation The test case needs to address the who, what, when, where and why. The missing question that needs to be answered is how. Rational Quality Manager includes a rich manual testing capability, allowing testers to create manual test scripts and use these to implement their test cases. Page 13

15 The manual test editor allows testers to capture textual test steps and embedded images that describe the scenario to be executed. In addition to this, Rational Quality Manager also allows testers to parameterise values in the test scripts allowing them to be executed with different datasets. While the manual testing capability is built-in to Rational Quality Manager, there are a number of integrations with test automation tools such as Rational Functional Tester, Rational Services Tester and Rational Performance Tester. Your IBM representative will be happy to address any questions around these or other automation tools. Keywords To enable test reuse, Rational Quality Manager uses keywords, which allow testers to share common steps between tests. This is effectively a reusable library of common business operations. For example, with a car insurance system, adding drivers details, accident history, endorsements are all ideally suited for reuse as they will be performed multiple times with varying data as part of testing this type of application. Keywords can simply be dragged from the keyword view into into the manual test editor and dropped at a suitable position in the manual test script. While a keyword looks to be part of the manual test, it isn't being copied into the test itself, the manual test script is calling out to the keyword. This substantially reduces maintenance time as a single keyword can be called many times, updating the keyword results in every script that calls it being updated instantly. The use of keywords also speeds up script creation due to the Page 14

16 level of reuse that can take place. Keywords are executed as part of the test script execution allowing the quality professional to verify the entire usage scenario. Keywords could be considered the building blocks used to construct a test script, because Rational Quality Manager has integrations with test automation tools. Some keywords may also be suitable for automation. This may be part of a gradual move towards test automation where business testers are supported by a member of the quality team using an automated testing tool or just a mechanism for making efficiency savings, automating manual activities such as a rolling back the database prior to performing a series of test runs. Rational Quality Manager includes a command line adapter that allows Rational Quality Manager to issue commands to the machine on which the adapter is running, such as executing batch, shell and SQL scripts and executables. Test Environments In order to verify that the software being developed will function correctly when it is deployed or installed in the end users environment, these test environments need to be defined based on the non-functional requirements of the software being developed and could include operating system support, service packs, web browsers and versions. Rational Quality Manager allows quality professionals to capture this test environment information and associate it with both the test plan and test cases to ensure coverage and verify the software being developed functions correctly in each environment. Testing every possible combination typically isn't feasible as the number of environments will grow exponentially with each new addition. Rational Quality Manager includes the ability to substantially reduce the number of permutations using techniques such as pair-wise testing to provide maximum breadth but limited depth. This ensures that each attribute is verified at least once. Attributes can be weighted to ensure there is a bias towards important attributes. Page 15

17 Test Execution Something that is sometimes overlooked when testing is the environment in which the test is being executed. This is especially common when the software being developed is an internal system to be deployed to an employees desktop. As this is a known entity, environmental attributes such as operating system, service pack and browser are often not considered relevant. However when a test is executed, it is always executed somewhere and this information needs to be recorded for accuracy of results. When executing a test case with Rational Quality Manager, it needs to be associated with a test environment, this combination of test case and environment is known as a Test Execution Record. Every test case will have at least one or more of these. The test execution record captures the result of executing the test case in that environment and is used in reporting to show that the software being developed functions correctly in each environment documented in the non-functional requirements. While the test execution record will show the latest result, the full history of previous results is also available. When executing a test execution record, the quality professional will select the record to run and click the button, this will then trigger the tests implementation to execute. Typically this will be a manual script but as has already been mentioned, could be an automated equivalent. Rational Quality Manager provides comprehensive test logs to allow both the tester and developer to re-examine any failures that occurred during execution and follow those steps that exposed the issue. Each step is a verification point with an expected result and can be passed or failed using the appropriate icon, either for pass or for fail. Manual test execution can also be paused and resumed to allow for executions that may be required to run across multiple days. For example, testing a batch process. Similar to the weighting idea used within the Rational Quality Manager test case, test results are broken down by the result type captured during the tests execution. Page 16

18 This allows Rational Quality Manager to report on quality at a more granular level so while the test execution record may have failed, we can see that 2 /3 of the functionality exercised is still working. Rational Quality Manager is able to report on results based on both TER count or weighting to provide a more accurate view of the quality of the software being developed. Rational Quality Manager provides the ability to group test cases into test suites and execute them in a specific sequence. Similar to test cases, the test suite should be associated with the test environment to ensure environmental coverage and verify the software being developed functions correctly in each environment. Test Lab Management Rational Quality Manager includes the ability to manage test lab assets such as desktops and servers. Part of this management includes a scheduling capability allowing members of the quality team to search and reserve a test machine in order to execute a series of tests. This allows the quality team to make maximise use the hardware available for their testing. Page 17

19 Part of any booking may well consist of having to set aside the time required to tailor the machine to fit in with your test environment needs. This could involve software installation, service pack updates or browser upgrades. An optional component of Rational Quality Manager provides the ability to execute a series of automation jobs on these test lab machines. These automations allow Rational Quality Manager to setup the test machines by automatically installing and removing any piece of software including applications, service packs or testing tools. Automations can be combined with test execution records or suites and called from a test schedule. Rational Quality Manager allows the quality professional to schedule a test run for a specified date and time, This is typically used in conjunction with automated test tools in order to perform out of office testing. By combining automations and automated tests in the same schedule, Rational Quality Manager is able to automatically build the test environment, install and configure the test automation tool, deploy the application under test and run a series of automated scripts. Once they are complete, Rational Quality Manager can remove the software leaving the machine in a clean state. Defect Tracking Rational Quality Manager has built-in defect tracking capabilities as well as integrations with the industry leading change management solutions such as Rational ClearQuest and Rational Team Concert as well as open source solutions. This allows quality professionals to raise defects either directly or as the result of a test failure, raising issues from the test log allows Rational Quality Manager to automatically establish traceability from the defect to the test log which then traces to the test case and requirement. Once a defect has been submitted it will follow an agreed workflow and will progress though a number of states until it can be marked as closed. Page 18

20 As the defect gets moved to each state, certain fields will become mandatory if information is required in order for the defect to progress. For example, to mark a defect as fixed a user may need to specify the resolution. Rational Quality Manager allows quality professionals to create queries against work items such as defects, requirements or tasks. This allows the user to filter the data and return the subset required. These queries can be personal to each user or shared across multiple users or teams. The results of these queries can then be added to the quality professionals dashboard to display the latest status of these items. Test Reporting The reporting engine supplied as part of Rational Quality Manager is known as Rational Common Reporting and is based on IBM Cognos technology. Rational Common Reporting is a separate download and is available from Jazz.net. It includes a large number of out of the box reports that can be customised if required to fit your individual needs. Your IBM representative will be more than happy to visit and demonstrate or discuss this functionality. Page 19

21 Customisation Rational Quality Manager offers customisation capabilities, depending on the asset, the level of customisation will vary. Test Plans and cases allow users to add user defined categories and sections whereas work items such as defects allow additional fields, tabs and work flows. Your IBM representative will be more than happy to visit and demonstrate or discuss this functionality. Next Steps Thank you for taking the time to evaluate IBM Rational Quality Manager, any questions resulting from this evaluation or any of the products mentioned, please ask your IBM representative who will be more than happy to help. In the hope that your Rational Quality Manager evaluation has proved successful, your IBM representative will be able to discuss the deployment of the proposed solution and provide guidance around any infrastructure requirements. A typical deployment of Rational Quality Manager may look similar to the below Page 20

22 As part of any proposed solution, your IBM representative will work in conjunction with you and our services team to develop an Implementation Plan, documenting the rollout and adoption of the chosen toolset. The services team will be led by an experienced engagement manager whose responsibility is to ensure the successful adoption of the IBM solution. The goal of any engagement is to make our customers self sufficient through the gradual implementation of tools, process, training and consultancy. Rational Quality Manager training will include: Essentials of Test Management with IBM Rational Quality Manager This course demonstrates the power of test management that is integrated with requirements, defect tracking, and test execution capabilities. You will learn how Rational Quality Manager provides a collaborative and customizable solution for test planning and workflow control. When this solution is combined with the ready-to-use tracking and metrics reporting, you can quantify how project decisions and deliverables impact and align with business objectives. Essentials of Manual Testing with IBM Rational Quality Manager The course helps you to create manual tests more quickly, streamline the execution of manual tests, and monitor manual test results. Additionally, you will learn how to increase the reuse of test assets, and how to use external test data with test scripts. Your IBM team is dedicated to ensuring your success using our software as the long term aim is the hope that you will be willing to highlight your success by becoming a reference. Additional Resources Additional articles and resources relating to IBM Rational Quality Manager can be found below IBM Rational Quality Manager in 3 minutes IBM Rational Quality Manager Demonstrations IBM Rational Quality Management ROI Calculator: IBM developerworks Quality Manager pages for hints, tools, tips and tricks: IBM Rational Quality Manager Online Help Page 21