A Framework to Manage and Evaluate Remote Software Testing Using the CMMI for Services Constellation. Dr. Aldo Dagnino

Transcription

1 A Framework to Manage and Evaluate Remote Software Testing Using the CMMI for Services Constellation Dr. Aldo Dagnino CMMI Technology Conference and User Group November 12-15, 2007 Hyatt Regency Tech Center, Denver, CO 1

2 Acknowledgements This work was developed for the SAS Institute, a private FORTUNE 100 Software Development Organization based in Cary, NC Process presented was used during the development and release of two SAS Products for the Manufacturing Solutions Group in

3 Agenda Introduction Geographically Distributed Product Development and Service Delivery Analysis of a Real-world Case Study Use of SAM PA for Product Development and Service Delivery Conclusions 3

4 Distributed Development Scenario at SAS Manufacturing Solutions Group Software development typically driven from the US Remote development organizations located in India System/Integration Software Testing performed in India Product Management owns product roadmap and is located in US Senior Management for Development organization located in US Consulting Group responsible to customize and implement software solutions in the field 4

5 Geographic Distribution Software (Product) Development Project Management Development Manager 2/3 Development Team 1/3 Development Team Software Verification (System and Integration Testing) Consulting Group Product Management Release Engineering R&D Senior Mgmt US India India US Improve Collaborative Development and Service Delivery 5

6 Findings CMMI Internal Appraisal 1 Strengths Organizational policy to manage external suppliers exist Supplier Agreements for COTS products are developed COTS products are evaluated against requirements Supply Chain Management handles the purchasing of commercial components for HW, SW and contractors All teams use common RE, CM, and Defect Tracking tools 6

7 Findings CMMI Internal Appraisal 2 Weaknesses No organizational policy/procedure to manage remote product development No organizational policy/procedure to manage remote service delivery (Testing) No formal collaboration agreements are established with remote teams Transition of work products (and services) provided by remote organization performed in informal manner 7

8 CMMI Practices Distributed Software Development RD PP SAM * * MA Distributed Software Verification (Service) SAM SVC PP SVC PMC SVC * * Note: SAM for Product Development and Service Delivery and MA will be the focus of this presentation 8

9 Process Area Relationships Stage 1 9

10 Process Area Relationships Stage 2 10

11 Process Area Relationships Stage 3 11

12 Process Area Relationships Stage 4 Responsible Product Development Partner MRS and PRS For Product Develop Project Plan Analyses And Reports Measurement Objectives RD PP MA Develop Collaboration Agreements Metrics and Analyses SAM Collaboration Agreement for Product Development (SAM) Collaboration Agreement for Service Delivery (SAM Service) Distributed Product Development Distributed VER 12

13 Process Area Relationships Stage 5 13

14 SAM Specific Goals/Practices - Appraisal Results SAM Process Area SG1 Establish Supplier Agreements SG2 Satisfy Supplier Agreements SP 1.1 Determine Acquisition Type SP 1.2 Select Suppliers SP 1.3 Establish Supplier Agreements SP 2.1 Execute the Supplier Agreement SP 2.2 Monitor Selected Supplier Processes SP 2.3 Evaluate Selected Supplier Work Products SP 2.4 Accept the Acquired Product/ Service SP 2.5 Transition Products/ Services Note: No procedures for collaboration/sub-contracting of products/services only for COTS 14

15 SAM Process Area -1- SG1 - Establish Supplier Agreements SP Determine Acquisition Type Acquisitions may be COTS from third-party vendors, components developed by internal or external partner, or services delivered by internal or external partner SP Select Suppliers Establish criteria for selection of partners and also list of preferred suppliers/collaboration partners SP Establish Agreements with Suppliers Establish formal agreements with suppliers and collaboration partners (service agreements, product development agreements, license agreements, etc) For internal partners the formal Supplier Agreement is a Collaboration Plan, which is part of the Project Plan 15

16 SAM Process Area -2- SG2 - Satisfy Supplier Agreements SP Execute the Supplier Agreement For internal partners the formal Supplier Agreement is a Collaboration Plan, which is part of the Project Plan SP Monitor Selected Supplier process For internal collaboration partners use internal release process SP Evaluate Selected Supplier Work Products This applies to internal developed components or services such as testing SP Accept the Acquired Product Services such as testing are also considered SP Transition Products Services such as testing are also considered 16

17 Remote Verification as a Service Activity - SAM SVC System and Integration Testing considered as a Service Delivery activity in the organization SAM SVC not Implemented in the past in the organization Service Delivery Capacity and Availability Management Problem Management Incident and Request Management 17

18 Sample Collaboration Agreement Templates Sample Templates derived from SAM PA to be distributed and discussed with attendees: Collaboration Agreement for Remote Product Development Collaboration Agreement Template for Remote Service Delivery (Software Testing/Verification) 18

19 MA Process Area 1 Measurement Objective To improve partner s satisfaction Measures Number of partner s complaints Party or stakeholder involved in collaboration can enter a complaint after a week of not having received response to an issue Level of severity of partner s complaints Low - first entry associated with a complaint Medium - second entry associated with a previous complaint High - more than two entries associated with a previous complaint 19

20 MA Process Area 2 Data Collection and Storage A partner/stakeholder enters a written complaint in the Complaint Spreadsheet available in the Project Common repository The Complaint Spreadsheet has several sections each regarding the identified type of collaboration The complaints are reviewed weekly at the Senior Management meetings Each manager is responsible to ensure any complaints are properly addressed Complaint Spreadsheet is maintained by Director of Development under CM 20

21 MA Process Area 3 Analysis of Measurement Data Histogram showing number of complaints clustered by severity level are developed by Director of Development Solutions 21

22 MA Process Area 4 Reporting of Measurement Data Histogram charts are presented at the end of each month and discussed at the Senior Management Meeting Any corrective actions are tracked to completion by Director of Development Solutions 22

23 Discussion on Measurement and Analysis 23

24 Sample of Complaints Sheet 2Q of

25 Sample Histogram of Complaints 2Q Complaints - R&D India to R&D US Number of Complaints High Medium Low Complaint Type 25

26 Conclusions 1 Geographically dispersed teams at SAS: Product Development System/Integration Testing System /Integration Testing viewed as Service With a low number of distributed projects, an informal method to collaborate was sufficient SAM CMMI PA needed as number of projects increased The practices of the SAM CMMI process area are successfully being used to manage both remote product development and service delivery 26

27 Conclusions 2 Including the CMMI MA PA helps monitoring effectiveness of process Essential to build a lean process Focusing on the most painful areas was important for buy-in Use of SAM process reduced level of frustration in remote sister organizations Resistance on process came from responsible partner Use of templates facilitated implementation of SAM process Metric was identified by members of the development and testing organizations 27

28 Author s Contact Information Dr. Aldo Dagnino ABB US Corporate Research Center Software Architectures Program Leader 940 Main Campus Drive Raleigh, NC USA [email protected] 28