Independent Test and Evaluation

Transcription

1 Independent Test and Evaluation ProPath Office of Information and Technology

2 Table of Contents Independent Test and Evaluation Process Maps... 1 Process: Independent Test and Evaluation... 3 Independent Test and Evaluation and Goals Goals... 4 Independent Test and Evaluation RACI Information... 5 Independent Test and Evaluation Process... 8 Process Activity Name: ITE-1 Review Testing Intake Assessment Results... 8 Process Activity Name: ITE-1-DEC01 Independent Testing Required?... 9 Process Activity Name: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting... 9 Process Activity Name: ITE-3 Perform Enterprise Systems Engineering Testing Process Activity Name: ITE-4 Perform SQAS Systems Integration Testing.. 11 Process Activity Name: ITE-5 Perform Operational Readiness Review Process Activity Name: ITE-5.1 Review ESE ERF Results Process Activity Name: ITE-5.1-DEC01 ORR Required? Process Activity Name: ITE-5.2 Create ESE Testing Checklist Process Activity Name: ITE-5.3 Submit/Resubmit ORR Process Activity Name: ITE-5.4 Execute ORR Acceptance Review Process Activity Name: ITE-5.4-DEC01 Execute ORR? Process Activity Name: ITE-5.5 Address ORR Acceptance Issues/Questions Process Activity Name: ITE-5.6 Execute ORR Process Activity Name: ITE-5.6-DEC01 Open Source Related? Process Activity Name: ITE-6 Provide to Open Source i

3 Independent Test and Evaluation Process Maps Independent Test and Evaluation home overview raci help Project Manager ITE-1 Review Testing Intake Assessment Results Independent Testing Required? Yes ITE-2 Conduct Enterprise Systems Engineering Testing Meeting No ITE-6 Provide to Open Source Yes Open Source Related? No Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst ITE-5 Perform Operational Readiness Review Enterprise Systems Engineering Test Analyst Systems Quality Assurance Service Independent Verification and Validation Manager ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Systems Integration Testing The links in this process map are inactive. Please scroll to view activity data. 1

4 Independent Test and Evaluation: ITE-5 Perform Operational Readiness Review home process overview raci help Project Manager ITE-5.3 Submit/Resubmit for ORR ITE-5.5 Address ORR Acceptance Issues/Questions Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst No ORR Required? ITE-5.1 Review ESE ERF Results Yes ITE-5.2 Create ESE Testing Checklist ITE-5.4 Execute ORR Acceptance Review Execute ORR? Rejected Accepted Issues/Questions ITE-5.6 Execute ORR The links in this process map are inactive. Please scroll to view activity data. 2

5 Process: Independent Test and Evaluation Overview: The process map for Independent Test and Evaluation cycles through the following process and review activities: ITE-1 Review Testing Intake Assessment Results ITE-1-DEC01 Independent Testing Required? ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Systems Integration Testing ITE-5 Perform Operational Readiness Review ITE-5.1 Review ESE ERF Results ITE-5.1-DEC01 ORR Required? ITE-5.2 Create ESE Testing Checklist ITE-5.3 Submit/Resubmit ORR ITE-5.4 Execute ORR Acceptance Review ITE-5.4-DEC01 Execute ORR? ITE-5.5 Address ORR Acceptance Issues/Questions ITE-5.6 Execute ORR ITE-5.6-DEC01 Open Source Related? ITE-6 Provide to Open Source 3

6 Independent Test and Evaluation and Goals Enterprise Testing Service (ETS), an organization within Enterprise Systems Engineering (ESE), works hand-in-hand with Product Development (PD) to provide an independent evaluation of development artifacts and product software. Both Systems Quality Assurance Service (SQAS) and ETS are two forms of evaluations that are not mutually exclusive. The type of evaluation required is dependent upon the results of risk-based analysis. The evaluations help the Office of Information and Technology (OI&T) management minimize risk of schedule delays, cost overrun, poor quality, and software failure. ETS provides: - Test environments on which software products can be tested - Independent evaluations of project artifacts and project software Services falling under the category of Test Environments include: - New test database in the ETS Test Center (ETSTC) - Miscellaneous issue for help with an ETSTC testing environment problem - Database refresh - Database backup/restore - User access request to add/reset/modify/deactivate user access to an existing ETSTC environment Test environment services are requested with an ETSTC Service Request. Services falling under the category of Independent Evaluations include: - Work product reviews of project artifacts - Test observation and validation of the operation and use of the software product - Various types of testing including legacy Patient Safety Issue (PSI) testing, risk-based independent testing, performance testing, and system integration testing Goals - Mitigate risk and provide feedback throughout the lifecycle - Ensure that products meet acceptable quality levels before promotion - Provide feedback for continuous improvement for process improvement - Continuously improve the likelihood that a quality product which meets user requirements is deployed on time and on budget - Ensure consistency in the testing methodologies and practices during Agile development - Validate that products are technically ready for incorporation into the VA infrastructure 4

7 Independent Test and Evaluation RACI Information The following describes the RACI information for this process: ITE-1 Review Testing Intake Assessment Results : Project Manager Accountable Role: Director, Project Management Division ITE-1-DEC01 Independent Testing Required? : Project Manager Accountable Role: Director, Project Management Division ITE-2 Conduct Enterprise Systems Engineering Testing Meeting : Project Manager Accountable Role: Director, Project Management Division ITE-3 Perform Enterprise Systems Engineering Testing : Enterprise Systems Engineering Test Analyst Accountable Role: Director, Project Management Division ITE-4 Perform SQAS Systems Integration Testing : Systems Quality Assurance Service Independent Verification and Validation Manager Accountable Role: Associate Director, Software Quality Assurance Service ITE-5.1 Review ESE ERF Results : Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering Informed Role: Project Manager 5

8 ITE-5.1-DEC01 ORR Required? : Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering ITE-5.2 Create ESE Testing Checklist : Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering Informed Role: Integrated Project Team, Project Manager ITE-5.3 Submit/Resubmit ORR : Project Manager Accountable Role: Director, Project Management Division ITE-5.4 Execute ORR Acceptance Review : Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering ITE-5.4-DEC01 Execute ORR? : Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering ITE-5.5 Address ORR Acceptance Issues/Questions : Project Manager Accountable Role: Director, Project Management Division 6

9 Informed Role: Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst ITE-5.6 Execute ORR : Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering Informed Role: Project Manager ITE-5.6-DEC01 Open Source Related? : Project Manager Accountable Role: Director, Project Management Division ITE-6 Provide to Open Source : Project Manager Accountable Role: Director, Project Management Division 7

10 Independent Test and Evaluation Process Process Activity Name: ITE-1 Review Testing Intake Assessment Results None ITE-1-DEC01 Independent Testing Required? The Project Manager reviews all testing intake assessment results, including Risk Analysis and Testing Scope Report (RATSR) or Testing Intake Analysis (TIA) closure . The Project Manager then incorporates ESE Enterprise Testing Service (ETS) Independent Testing and/or Systems Quality Assurance Service (SQAS) independent testing services into project plans and schedules. Artifacts Used ESE Risk Analysis and Testing Scope Report (RATSR) Initial ESE Test Plan Requirements Specification Document (RSD) System Design Document (SDD) Artifacts Created Independent Testing Requirements (None, SQAS Testing, ESE Testing, Both) Project Manager Tools and Websites Capacity and Performance Engineering SharePoint site Enterprise Testing Service Portal Standards None Listed More Info ESE ETS and SQAS evaluations are not mutually exclusive and either one or both types of testing may be needed. 8

11 Process Activity Name: ITE-1-DEC01 Independent Testing Required? ITE-1 Review Testing Intake Assessment Results If Yes, the following activities are accomplished concurrently: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Systems Integration Testing If No, the next activity is ITE-4-DEC01 ORR Review Required? Note: There is a decision dependency that determines the next activity to be performed. If the Project Manager determines independent testing is required, the following activities are accomplished concurrently: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Systems Integration Testing If the Project Manager determines no independent testing is required, the next activity is ITE-4- DEC01 ORR Review Required? Project Manager Process Activity Name: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting Note: This activity is performed concurrently with: ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Integration Testing ITE-1-DEC01 Independent Test Required ITE-5 Perform Operational Readiness Review 9

12 The Project Manager conducts a kick off meeting with Enterprise Systems Engineering (ESE) Enterprise Testing Service (ETS) in order to plan and coordinate the delivery of testing services identified for the product in the ESE Risk Analysis and Testing Scope Report (RATSR) (completed in Project Planning Process). The ESE Testing Checklist for the project is completed prior to the meeting. Artifacts Used ESE Risk Analysis and Testing Scope Report (RATSR) Initial ESE Test Plan Artifacts Created Artifact Review Agenda and Minutes Project Manager Tools and Websites Enterprise Testing Service Portal Standards Enterprise Testing Service Testing Guide More Info None Listed Process Activity Name: ITE-3 Perform Enterprise Systems Engineering Testing Note: This activity is performed concurrently with: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-4 Perform SQAS Systems Integration Testing ITE-1-DEC01 Independent Testing Required? ITE-5 Perform Operational Readiness Review 10

13 The Enterprise Systems Engineering (ESE) Test Analysts execute Independent Testing. The scope and type of Independent Testing is defined in the Project Risk Assessment and Testing Scope Report (RATSR) completed in the Project Planning Process. Services include but are not limited to: - Interoperability/Integration Testing - Patch Release Testing for Patient Safety Issues - Requirements Validation - Test Observation and Validation - Performance Testing Artifacts Used ESE Risk Analysis and Testing Scope Report (RATSR) ESE Testing Checklist Project Documents Determined by Type of Test Artifacts Created Enterprise Testing Service Analysis Report Enterprise Testing Service Performance Testing Information Request Final ESE Testing Test Plan Enterprise Systems Engineering Test Analyst Tools and Websites Enterprise Testing Service Portal Standards Enterprise Testing Service Testing Guide More Info The Service Delivery and Engineering (SDE) Program Administration Office (PAO) Process Site provides additional information on the SDE PAO including organization charts, Governance Board Charter, and an overview presentation on PAO Services. Process Activity Name: ITE-4 Perform SQAS Systems Integration Testing Note: This activity is performed concurrently with: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting 11

14 ITE-3 Perform Enterprise Systems Engineering Testing ITE-1-DEC01 Independent Test Required ITE-5.1 Review ESE ERF Results In accordance with VA Directive 4900, the Systems Quality Assurance Service (SQAS) Independent Verification and Validation (IV&V) Manager ensures the required Systems Integration Testing (SIT) and Quality Assurance Activities are performed on software if it impacts VA financial systems of record or software that interfaces with such a financial system. SIT verifies the accuracy of the delivered software product and determines whether the product built satisfies the customers requirements based on functional, system and design specifications. SQAS develops the IV&V Software Quality Assurance Plan (SQAP) and IV&V SIT Plan, requirements-based test plans, test cases and test scripts with a focus on internal controls including general and application controls. Internal controls form the core of the Federal Government s financial management standards. Test scripts are executed against project software deliverables to verify functionality, including interfaces, and to perform regression testing, or verify correction of defects ( bug fixes ). SQAS issues certification for production implementation and verifies the installation of software configurable items. The delivered software module or release is verified using configurable item attributes. Artifacts Used Business Requirements Document Defect Report Interface Control Document Master Test Plan Product Build Product Documentation Requirements Specification Document Requirements Traceability Matrix Software Design Document Software Quality Assurance Plan System Build System Design Document System Integration Plan Test Cases/Test Scripts Test Readiness Review Report Test Summary Report Artifacts Created IV&V SIT Defect Report IV&V SQAS Test Report IV&V Test Cases 12

15 IV&V Test Scripts SQAS Certification Software Quality Assurance Plan Updated System Integration Plan Updated Systems Quality Assurance Service Independent Verification and Validation Manager Tools and Websites IBM Rational Quality Manager IBM Rational RequisitePro IBM Rational ClearCase Standards Federal Manager's Financial Integrity Act of 1982 IEEE Standard 829 for Software and System Test Documentation IEEE Standard 1012 for System and Software Verification and Validation OMB Circular A-123, Management s Responsibility for Internal Control OMB Circular A-127, Financial Management Systems (revised) Software Configuration Management Plan Standard VA Directive 4900, Internal Controls for Financial and Financial Interfacing Automated Information Systems More Info The Federal Managers Financial Integrity Act of 1982 requires that VA systems and their interfaces comply with quality assurance processes that support functional, technical and internal control requirements. Process Activity Name: ITE-5 Perform Operational Readiness Review ITE None Listed The process map for ITE-5 Perform Operational Readiness Review cycles through the following dependent activities: - ITE-5.1 Review ESE ERF Results - ITE-5.1 ORR Required? 13

16 - ITE-5.2 Create ORR Checklist - ITE-5.3 Submit/Resubmit for ORR - ITE-5.4 Execute ORR Acceptance Review - ITE-5.4-DEC01 ORR Rejected? - ITE-5.4-DEC02 ORR Accepted? - ITE-5.5 Address ORR Acceptance Issues/Questions - ITE-5.6 Execute ORR Process Activity Name: ITE-5.1 Review ESE ERF Results Note: The following activities are performed concurrently: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Systems Integration Testing ITE-5.1-DEC01 ORR Required? The Enterprise System Engineering Capacity and Performance Engineering (ESE CPE) Operational Readiness Review (ORR) Analyst analyzes the disposition of the Enterprise Systems Engineering Registration Form System Characteristics - Project Attributes Form (ERF SC PA) to determine if ESE CPE Operational Readiness Review (ORR) is required. Artifacts Used ESE Registration Form Artifacts Created ESE Capacity Planning and Engineering (CPE) Services ORR Analysis Report Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Tools and Websites Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site Enterprise Systems Engineering (ESE) ORR Checklist Repository 14

17 ESE Registration Form Portal Standards Enterprise Systems Engineering (ESE) Operational Readiness Review Reference More Info Files used for ORR are uploaded as zip files to the ESE ORR Checklist. Process Activity Name: ITE-5.1-DEC01 ORR Required? ITE-5.1 Review ESE ERF Results If YES, ITE-5.2 Create ESE Testing Checklist OR If NO, process ends. The Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst determines if ESE CPE Operational Readiness Review (ORR) is required. Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Process Activity Name: ITE-5.2 Create ESE Testing Checklist ITE-5.1-DEC01 ORR Required? ITE-5.3 Submit/Resubmit for ORR The Enterprise Systems Engineering (ESE) Capacity Planning and Engineering (CPE) Operational Readiness Review (ORR) Analyst creates the ESE Testing Checklist as a template for the project team to complete if ESE CPE ORR is required for the project release. Artifacts Used ESE Registration Form 15

18 Artifacts Created ESE Testing Checklist Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Tools and Websites Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site Enterprise Systems Engineering (ESE) Testing Checklist Repository ESE Registration Form Portal Standards Enterprise Systems Engineering (ESE) Operational Readiness Review Reference Enterprise Systems Engineering Testing Intake Assessment Form User Guide More Info None Listed Process Activity Name: ITE-5.3 Submit/Resubmit ORR ITE-5.2 Create ESE Testing Checklist ITE-5.4 Execute ORR Acceptance Review The Project Manager completes the ESE Testing Checklist and submits for ESE CPE ORR if required for the project release. Once all non-production testing is completed on the build that deploying to Initial Operating Capability (IOC) production or Release (if IOC is not applicable). If the project team previously submitted for ESE CPE ORR and the ESE CPE ORR was rejected, the project team resubmits for ESE CPE ORR. Artifacts Used ESE Testing Checklist 16

19 Artifacts Created Updated ESE Testing Checklist Project Manager Tools and Websites Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site Enterprise Systems Engineering (ESE) ORR Checklist Repository ESE Registration Form Portal Standards Enterprise Systems Engineering (ESE) Operational Readiness Review Reference Enterprise Systems Engineering (ESE) Operational Readiness Review Guide More Info None Listed Process Activity Name: ITE-5.4 Execute ORR Acceptance Review ITE-5.3 Submit/Resubmit for ORR ITE-5.4-DEC01 Execute ORR? The assigned ESE CPE ORR Analyst for the project releases executes an ESE CPE ORR Acceptance Review. The ESE CPE ORR Analyst analyzes the submitted project release artifacts against the published ESE CPE ORR Acceptance Criteria in order to determine the ESE CPE ORR disposition: - Rejected: The submitted ESE CPE ORR is grossly deficient (requirements, requirement traceability, testing results, required items are not provided, etc.). The ESE CPE ORR is rejected and the project team is notified. - Acceptance Issues: The submitted ESE CPE ORR is not meeting some of the published ESE CPE ORR Acceptance Criteria, but does not qualify as a candidate for rejection. The ESE CPE ORR analyst notifies the project team of the issues and/or questions and the ESE CPE ORR is moved to an acceptance issues status. 17

20 - Accepted: The ESE CPE ORR meets all the published ESE CPE ORR Acceptance Criteria. The project team is notified and the ESE CPE ORR is moved to the accepted status. Artifacts Used ESE Testing Checklist Artifacts Created ESE CPE ORR Acceptance Issues/Questions ESE CPE ORR Accepted ESE CPE ORR Rejection , or Updated ESE Testing Checklist Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Tools and Websites Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site Enterprise Systems Engineering (ESE) ORR Checklist Repository ESE Registration Form Portal Standards Enterprise Systems Engineering (ESE) Operational Readiness Review Reference Enterprise Systems Engineering (ESE) Operational Readiness Review Guide More Info None Listed Process Activity Name: ITE-5.4-DEC01 Execute ORR? ITE-5.4 Execute ORR Acceptance Review If Accepted, ITE-5.6 Execute ORR Or If Rejected, ITE-5.3 Submit/Resubmit for ORR Or 18

21 If Issues/Questions, ITE-5.5 Address ORR Acceptance Issues/Questions The Enterprise Systems Engineering Capacity Planning and Project Manager Engineering Operational Readiness Review Analyst determines if the project is ready for Operational Readiness Review (ORR). There are 3 decisions that can occur: 1. If accepted, the ESE CPE ORR meets all the published ESE CPE ORR Acceptance Criteria. The project team is notified and the ESE CPE ORR is moved to accepted status and the next activity is ITE-5.6 Execute ORR. 2. If rejected, the submitted ESE CPE ORR is grossly deficient in that requirements, requirement traceability, testing results, required items, etc. are not provided. The ESE CPE ORR is rejected and the project team is notified and the next activity is ITE-5.3 Submit/Resubmit for ORR 3. If acceptance issues are found, the submitted ESE CPE ORR is not meeting some of the published ESE CPE ORR Acceptance Criteria, but does not qualify as a candidate for rejection. The ESE CPE ORR analyst notifies the project team of the issues and/or questions, the ESE CPE ORR is moved to an acceptance issues status and the next activity is ITE-5.5 Address ORR Acceptance Issues/Questions. Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Process Activity Name: ITE-5.5 Address ORR Acceptance Issues/Questions ITE-5.4-DEC01 ORR Decision? ITE-5.4 Execute ORR Acceptance Review The Project Manager addresses the Enterprise Systems Engineering (ESE) Capacity and Performance Engineering (CPE) Operational Readiness Review (ORR) Acceptance Issues/Questions and responds to the assigned ESE CPE ORR Analyst. Artifacts Used ESE CPE ORR Acceptance Issues/Questions 19

22 Artifacts Created ESE CPE ORR Acceptance Issues/Questions response Project Manager Tools and Websites Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site Enterprise Systems Engineering (ESE) ORR Checklist Repository ESE Registration Form Portal Standards Enterprise Systems Engineering (ESE) Operational Readiness Review Reference Enterprise Systems Engineering (ESE) Operational Readiness Review Guide More Info None Listed Process Activity Name: ITE-5.6 Execute ORR ITE-5.4-DEC01 Execute ORR? Process Ends The Enterprise System Engineering Capacity and Performance Engineering (ESE CPE) Operational Readiness Review (ORR) Analyst executes the ORR for the accepted project release. The ORR analyzes and evaluates key technical areas pertinent to ESE, within the context of the service delivery of a high-quality product, especially: requirements definition, design, traceability, sufficiency of testing, operability (installation and back out), capacity, system resource consumption (including bandwidth), and performance. Artifacts Used ESE Testing Checklist 20

23 Artifacts Created ESE Capacity Planning and Engineering (CPE) Services ORR Analysis Report Updated ESE Testing Checklist Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst Tools and Websites Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site Enterprise Systems Engineering (ESE) ORR Checklist Repository Standards Enterprise Systems Engineering (ESE) Operational Readiness Review Reference More Info None Listed Process Activity Name: ITE-5.6-DEC01 Open Source Related? ITE-5.6 Execute ORR If Yes, ITE-6 Provide to Open Source If No, None Note: There is a decision dependency that determines the next activity to be performed. If open source related, the next activity is ITE-6 Provide to Open Source. If not open source related, the process ends. Project Manager Process Activity Name: ITE-6 Provide to Open Source ITE-5.6-DEC01 Open Source Related? 21

24 None The Project Manager submits a list of required software code, system builds, and artifacts as described in the Open Source Submission Checklist to OI&T PD Open Source, Configuration & Tools Management Division via the mail group OIT Open Source Software Delivery or OSSOFT@va.gov. Artifacts Used Open Source Submission Checklist Product and Project Artifacts Software Code System Build Artifacts Created Updates to Open Source Community Project Manager Tools and Websites IBM Rational ClearCase Standards Open Source Code in Flight Communications Strategy Guide More Info The Open Source Submission Checklist lists different artifacts and/or system builds/codes that are to be submitted at certain intervals or milestones to OI&T PD Open Source, Configuration& Tools Management Division, as a product is being built. As project managers are developing their project plans, providing code and artifacts to OI&T PD Open Source, Configuration& Tools Management Division should be considered and incorporated into the plan both based on specific milestones and time intervals basis, as specified in the Open Source Submission Checklist. END OF PROCESS 22