STC - 2013 Test Report Dashboard Art of Showcasing data graphically, dynamically

STC - 2013 Test Report Dashboard Art of Showcasing data graphically, dynamically Prepared by: Indium Software India Ltd. Name : Poornima Gopalan & Vishnupriya B Email : poornima.g@indiumsoft.com vishnupriya.b@indiumsoft.com Contact # : 91-98409 81449, 91-90925 25934

Test Report Dashboard Art of Showcasing data dynamically, graphically 1 TABLE OF CONTENT 1 ABSTRACT... 2 2 WHY WE NEED... 2 3 THE IMPORTANCE... 2 4 REPORT TYPES... 3 4.1 DAILY / WEEKLY / MONTHLY STATUS REPORT... 4 4.1.1 Contents... 4 4.1.2 Report Designing... 4 4.1.3 Benefits... 6 4.1.4 Recipients... 6 4.2 PRODUCTIVITY / PROJECT MANAGEMENT METRICS REPORT... 6 4.2.1 Contents... 6 4.2.2 Report Design... 7 4.2.3 Benefits... 8 4.2.4 Recipients... 8 4.3 OVERALL TEST SUMMARY... 9 4.3.1 Contents... 9 4.3.2 Report Design... 9 4.3.3 Benefits... 10 4.3.4 Recipients... 10 5 CONCLUSION... 11 6 REFERENCES... 11 7 AUTHOR BIOGRAPHY... 11

Test Report Dashboard Art of Showcasing data dynamically, graphically 2 1 Abstract One of the important facets of software development life cycle is testing. Software testing is an area that is being considered and given utmost importance in the world of fast changing technology. There are various stages at which testing are done to ensure quality of delivery. Reports are the basis for many processes used to manage and communicate results from business operations. In business today, the reporting process is often incredibly complex. It can include reports derived directly from information systems, custom reports developed from data warehouses, and reports based on spreadsheets that compile various system and manual inputs, among many others. Most key reports not only assist corporations in understanding pertinent financial information, but also are critical components of the control framework of the organization. 2 Why we need Visibility, Accountability & Transparency (VAT) are essential to any project/product team at different levels such as Project/Product level, Team level, Task level and also at the Team members level. An effective status report is the only way that can make sure that these attributes are met We do send reports on a day to day basis to communicate the accomplishments / tasks performed for the day, however as a product owner or product team member do we get a consolidated list of these at will? The Answer is a big NO. The team has to either go through a bunch of status reports or wait for the test team to consolidate and send a weekly status report / test summary report. Our solution is to inculcate all the objectives of a daily / weekly / summary report into a single report which not just reduces time to prepare but also helps the project team to have access to all the data that they seek for at their will. 3 The Importance Is it sufficient to give a single line certification that, the product is ready/not ready for implementation? A detailed test reporting should bring in transparency to test coverage / effectiveness / defect metrics / test confidence which the customer would appreciate most. This could help the customer to look for alternate solutions sometimes An effective test report should highlight Test Scope for release / sprint / product Test Coverage Test Effectiveness Test Metrics Strengths, Limitations and Recommendations Impediments Key Issues with solutions Other aspects such as performance, platform dependence etc. Test Confidence

Test Report Dashboard Art of Showcasing data dynamically, graphically 3 4 Report Types Testing teams send out various reports at different phases of the STLC, Planning Designing It is enough to communicate with the rest of the project teams, when a test plan is created or when a major change is made to it based on the overall product / project plan devised Reports in these stages will highlight information on the High level test scenarios (HLTS) / Low Level Test cases (LLTC), Traceability Matrix (RTM), Test Data generation and other activities to all the project team stakeholders. These will not only let them know about the progress of the task but also signal the teams that need to review and provide signoff on the artifacts, that they are up next Execution is the phase of a project when the testing team is the primary focus positively and negatively. Execution The test execution stage consists of: Setting up the test environment as detailed in the test case specification Selecting and running relevant tests from the test case spec Recording test results Recording and managing defects Final Reporting of the test execution phase A typical day during a test cycle is not done, unless the daily status report is sent out. In some teams, they could agree on a weekly report, but having it sent daily is the norm. It is also not uncommon to have a status meeting everyday (or week) to present the QA team s status to the concerned parties. Hence, the mode of a status report can be: Email/document Meeting/presentation The following are the different types of test reports: Daily / Weekly Status Report Productivity / Project Management Metrics Report Monthly Summary / Release (or) Sprint summary Report Overall Release / Test Summary Report

Test Report Dashboard Art of Showcasing data dynamically, graphically 4 4.1 Daily / Weekly / MonthlyStatus Report Status report templates are used by teams needing frequent check-ins against progress. Product development and project management teams often use daily status reports (sometimes in the form of a scrum) to keep everyone coordinated and on track toward team goals. During the execution phase, test case results and any defects will need to be recorded in order to track and manage the test progress until all planned tests are completed or the final acceptance milestone has been reached. This information is often stored via a test management tool such as Quality Center or an Excel spreadsheet such as the Test Execution Dashboard which contains the following 4.1.1 Contents On a day to day basis, we provide information on the four parameters listed below What did we do today? What is for tomorrow? Impediments we faced with resolutions (if any) Dependencies In addition, what we propose is to include contents of what exists generally in the test summary reports such as Test Scope highlighting the features (or) areas in scope for the release Test Approach Depicts the strategy planned for the release / test cycle Test Deliverables Lists all the deliverables to be sent for review / submission Test Coverage Indicating the progress made everyday (Designing / Execution stage) Test Effectiveness Indicating status of each of the LLTC (Execution stage only) Defect Metrics - By Status / Priority / Severity / Module (or) Verticaletc. to name a few Key Issues with resolutions This helps to conduct postmortem on what went wrong and how we resolved the issue and serves as data point for future reference. E.g. capturing environment downtimes with reasons Test Resource Productivity Capturing information on how much time was spent by each resource on which task / activity to capture the root cause analysis on any schedule deviation or effort variance The same report can include additional information such as Build Metrics capturing the number of release number or version, number of features implemented, number of bugs fixed etc. to have more insight on the number of builds worked on by the test team for a Product / Project Release 4.1.2 Report Designing It becomes difficult when all the above mentioned has to be represented into a single component and still not be looking as a troublesome process from an understanding perspective. The below snapshot depicts how to represent the status report effectively and efficiently in MS Excel 2010 using some of the key features such as Pivot tables & Slicers which help us to generate data dynamically. Header: Includes the Client name & the Reporting Period, the current date is listed below, on the right side of the header is a synopsis of the test approach Middle Section: Includes the day wise activities along with the areas in scope & resources used. The right pane of the report includes graphical representations of the overall defects in the product / project along with the slicers which allow us to filter and view the graphs dynamically based on the selection. For instance the user can choose to view the current list of defects that are critical and active. This can also support applying multiple filters This section also includes areas for deliverables in scope, action items/ Impediments and also helps to capture key issues with resolution notes

Test Report Dashboard Art of Showcasing data dynamically, graphically 5 Fig.1 Daily Status report Dash board Bottom Section: Includes Test design metrics on how many High level test cases are designed by module or vertical and how many low level test cases were drafted for the HLTS identified. This can include the coverage / effectiveness graphs in the execution phase and can have slicers for the users to filter and view the report Fig.2 Test Case Completeness by Module Fig.3 Test Coverage & Test Effectiveness

Test Report Dashboard Art of Showcasing data dynamically, graphically 6 Test Coverage:A measure of the proportion of a program exercised by a test suite, usually expressed as a percentage. This will typically involve collecting information about which parts of a program are actually executed when running the test suite in order to identify which branches of conditional statements which have been taken. Test coverage matrix determines the test completeness. It measures the percentage of test execution details with respect to the no. of high level cases designed Test Effectiveness: This is a metric to captures the effectiveness of the test cases executed based on the number of passes/ failures of the documented test scenarios. This determines the test confidence on the application. Execution & Coverage reports answer the following questions: How much work is complete, how much work is left, and is progress being made as expected? Which module / vertical have high numbers of failed points or test case execution records? Which module / vertical plans have high numbers of defects? Which systems have high numbers of failed points or test execution records? Which test case categories have high numbers of failed points? Monthly Status reports shall include consolidated weekly sheets, wherein we pull in data from different weeks and create a data sheet from which project team can pull information on any specific week or release. 4.1.3 Benefits The objective of capturing this data on a day to day basis is to Save timeby not having to spend repeated efforts of preparing multiple reports through consolidation of your daily tasks. Instead updating this sheet on a daily basis converts the same into a weekly status report with few additional sections and minimal effort Enhance Tracking mechanism by bring in Transparency & Visibility on the progress made from the designing stage to the execution stage and thereafter as well Improve Communication by communicating risks / issues on a regular basis Improves efficiency by analyzing the productivity / root cause analysis on the delays 4.1.4 Recipients Generally, the recipient of this email/report is generally the manager, also the team members can be CC ed in some cases this depends on the communication protocol the team follows but we recommend this to be made available in a repository (Cloud / Server) for any of the project stakeholder to have access to the document at his/her own will. 4.2 Productivity / Project Management Metrics Report The core objective of this report is to get insight on the productivity of the team members on various tasks /activities performed in different stages of the STLC which can be used to transform how we can manage resources and derive sustainable growth in future which can help the organization. 4.2.1 Contents In today s fast world, getting timesheet information filled is considered as strenuous ask, developing an internal application or procuring one from market, customizing, maintaining is even more challenging. To avoid all such we developed a simple 4 step process to record the task / activities executed by the team members with the timelines. The productivity report will include the following Day wise task/ activities with the time spent Schedule Deviation with RCA Effort Variance with RCA Milestones Achieved Upcoming Milestones Top Risks with mitigation plan

Test Report Dashboard Art of Showcasing data dynamically, graphically 7 4.2.2 Report Design Summary sheet will include project specific information such as Lead, customer contact,domain, start & end date along with resources allocated by module / verticals. The productivity of the resources will be captured as a consolidated list against modules if there are multiple resources working on the same module. Additional information such as the number of billable / shadow resources and new resource request shall also be comprised as part of this section Fig.4 Project Management Metrics Dashboard The second half of the report includes the Delivery related metric such as capturing the planned deliveries against the actual deliveries made and includes information on the root cause of the deviances (if any) Some of the common RCA s are as listed below Fig.5 Effort Variance RCA Change Request Coding Issue Coding Standard Issue Analysis Issue Availability Change in Priority Configuration CR Data Issue Design Issue Duplicate Issue Environment issues Estimation Framework Issue Incompetency Not an Issue Planning Skill Unclear Requirement

Test Report Dashboard Art of Showcasing data dynamically, graphically 8 The second sheet in this report includes the detailed productivity hours per resource as depicted below. This serves as an input to the productive hour s column in the summary sheet Fig.6 Productivity Hours Table The last section of the summary report will include additional metrics specific to the team members and their contributions towards the project on weekly basis or activity wise Fig.7 Weekly basis Metrics 4.2.3 Benefits Helps organization to identify the effective productivity of resources Helps in analyzing areas where the project team had spent long hours and helps improve resource optimization and planning for future RCA helps the customer and the project team in determining the milestones for future releases Improves test estimation for future projects with similar domain/ technology Helps customer to realize ROI for any kind of delivery engagement esp. T& M projects as this brings in transparency on the output of the individuals in the team 4.2.4 Recipients The recipient of this email/report is to all project stake holders Internal & Customer. This report can also be maintained in the repository (Cloud/ Server).

Test Report Dashboard Art of Showcasing data dynamically, graphically 9 4.3 OverallTest Summary Test summary report will be different for different kind of testing. If the project is risk based testing then the report should contain the test coverage and unresolved defects against the product risks, and if it is requirements based testing then the test summary report should measure coverage against the requirements. In addition to test coverage and unresolved defects test summary reports should also contain test strategy, test objectives and overall result of test effort. 4.3.1 Contents Test Summary Report Includes: Objective Test Scope Variances Comprehensive assessment Evaluation Executive Test Summary Our Challenges & Findings Project Metrics Project Deliverables Summary of activities Summary of results Next Steps Approvals 4.3.2 Report Design Test Summary report can be best showcased in MS presentation. It shall contain following sections along with pictorial representations. Objective: Includes the testing focused areas, system requirements and features in scope for the testing. Fig.8 Test Summary - Objective Test Process: Includes the process that the QA team followed to streamline the application under test and to enhance the test effectiveness. Also includes analyzing of test Results, to determine the effectiveness of testing that QAperforms Challenges and Key findings: Includes the challenges we faced during this Sprint/Release. The Major issues of this Sprint/Release will come under Key findings.

Test Report Dashboard Art of Showcasing data dynamically, graphically 10 Value Adds:Includes the recommendations / suggestions or any other extra features beyond the standard expectations that might have given good ROI. This can also include following: Total number of defects raised No. of recommendations given Defect Metrics:Includes the overall project defect metrics in a pictorial representation based on following categories, in the same format as DSR i. Status ii. Priority iii. Severity iv. Bug Type v. Module / Component wise vi. Also, the statistics of defects based on two categories can be given as in following graphical representation Fig.9 Test Summary Resolution vs. Priority 4.3.3 Benefits There are many benefits that can be gained by using Test Reports. They are: Clear visibility of work in progress Reduction of time spent Greater consistency and repeatability Objective assessment Ease of access to information about tests or testing Helps development team to achieve better quality in future releases Helps customers with key decision making with respect to release of products 4.3.4 Recipients The recipient of this email/report is to all project stake holders Internal & Customer. This report can also be maintained in the repository (Cloud/ Server).

Test Report Dashboard Art of Showcasing data dynamically, graphically 11 5 Conclusion Test reporting has no end to the value additions that one could provide with good report dashboard. But it is prominent to trail test report from start of any project until its closure A dynamical and accurate test report dashboard helps its recipients obtain better visibility for key decision with respect to release of products Dynamic test reports can be showcased well in MS Excel along with graphical data All future risks, mitigation & contingency plan can be arrived at by tracking pertinent test reports. Future milestones for any project specific proceedings also stands as easy to arrive with virtuous report dashboard 6 References http://istqbexamcertification.com/what-is-test-coverage-in-software-testing-its-advantages-anddisadvantages/ http://www.softwaretestingmentor.com/test-deliverables/test-execution-reports/ 7 Author Biography Poornima Gopalan has an overall IT experience of 9 years on delivering IT solutions on several technologies over different domains through Software testing. She is a Quality Analyst in Indium Software and was associated previously with Cognizant Technology Solutions. Her current work includes coordinating and managing multiple test teams to deliver quality at its best. She has passion in training testers to go beyond test functionality and coverage. Her interests also include test reporting and metrics. She is also passionate about Professional Management Professional and a member of pmhub.net Vishnupriya comes with 2.5 years of experience and ISTQB-BCS certified professional. She started her career as Assistant Engineer QA in SmarTek Consultancy Services and worked in various projects related to mobile technology. Currently she is working in Indium Software as Test Engineer for mobile application testing. She did her Bachelor degree in Electrical and Electronics from Anna University, Tiruchirappalli. Her hobbies are listening music and crafts work.