Construction Management System (CMS) Deliverable Review Process

Transcription

1 Construction Management System (CMS) Deliverable Review Process State of California Department of Transportation Division of Construction December 21, 2009 Version 1.0

2 Approvals Name: Title: Mark Leja Project Sponsor Name: Title:

3 TABLE OF CONTENTS 1 INTRODUCTION BACKGROUND DOCUMENT OBJECTIVES RESOLUTION OF CONFLICTS OR INCONSISTENCIES SUMMARY OF THE REVIEW PROCESS PROCESS SCOPE IMPORTANT FEATURES OF THE DELIVERABLE REVIEW PROCESS IDENTIFICATION AND NOTIFICATION OF DELIVERABLE REVIEWERS DELIVERABLE REVIEW PERIOD COMMENTS ON DRAFT DELIVERABLES FINALIZING DELIVERABLES DELIVERABLE ACCEPTANCE CRITERIA FORMAL DELIVERABLE ACCEPTANCE NEXT STEPS... 8 APPENDICES APPENDIX A: RFP CONTRACTOR DELIVERABLE REQUIREMENTS APPENDIX B: LIST OF DELIVERABLES SUBJECT TO THE DELIVERABLE REVIEW PROCESS APPENDIX C: DELIVERABLE REVIEW PROCESS FLOW CHART APPENDIX D: DELIVERABLE COMMENTS MATRIX CMS Project Page i Deliverable Review Process

4 REVISION HISTORY Version Date Description Author Number /21/09 Final Jane Zhang CMS Project Page ii Deliverable Review Process

5 1 INTRODUCTION 1.1 Background The Department of Transportation (Department), Division of Construction develops, implements, and maintains enterprise construction support Information Technology (IT) systems related to construction engineering and contract administration. Currently the Department s construction contracts are being managed by various systems. The primary enterprise construction system is the Contract Administration System (CAS). CAS is a legacy mainframe computer system using COBOL programming language, and has been used by the Department since Due to the age and technology constraints of CAS, it has become increasingly difficult to keep pace with legislative changes and significant growth of construction programs. CAS is not able to support the informational needs of the construction programs, and there is an increasing risk of system failure. In addition, the Department does not have a single, reliable system with which to manage all construction project activities. Information is in multiple systems, including small desktop databases. In some cases, the most current data is contained in paper files within the districts. Since CAS does not store all project records, the Resident engineers (REs) at project sites are responsible for manual record keeping and researching volumes of paper when post-construction claims, arbitration or audits occur. Manual handling, ever-increasing paper documents, and files are labor intensive, error prone, and difficult to access, thus causing additional program cost and time with less product quality. To address the business problems, the Division of Construction has begun an effort to replace its current legacy CAS with a new Construction Management System (CMS). The proposed CMS solution is a Web-based centralized construction management application that will be accessible by all Department construction staff, contactor staff, and internal and external stakeholders. The total one-time project cost as planned over an 8-year period is approximately $20.5 million with full implementation expected to be completed in September The CMS Feasibility Study Report (FSR) was approved by the Department of Finance (now The Office of the State Chief Information Officer (OCIO)) on January 10, The first CMS Special Project Report (SPR) was approved on April 1, 2008 by the OCIO, and the latest CMS SPR is currently in the review and approval process. 1.2 Document Objectives This document presents the Deliverable Review Process for the CMS project. It specifically documents the process that will be used to submit, track, review, update, finalize, and approve deliverable documents provided by the CMS prime contractor (contractor). CMS Project Page 1 Deliverable Review Process

6 1.3 Resolution of Conflicts or Inconsistencies The CMS Request for Proposal (RFP) and the subsequent contract contain contractor deliverable requirements for the contractor to submit deliverables. The process described in this document is meant to supplement and/or provide further detail on the requirements provided in the CMS RFP (RFP DOT ) and any subsequent contracts, amendments, or agreements with the contractor. Should any inconsistencies or conflicts exist between this document and either the RFP or contract, amendments, or agreements, the RFP or contract, amendments, or agreements will take precedence. Refer to Appendix A for the Contractor Deliverable Requirements excerpted from the RFP. CMS Project Page 2 Deliverable Review Process

7 2 SUMMARY OF THE REVIEW PROCESS The CMS project deliverable submittal and review process is provided in the following subsections. 2.1 Process Scope This process pertains to the submittal of contractor deliverables which are documents, rather than automated tools or project software. A preliminary list of the deliverables subject to this process is identified in Appendix B. It is important to note that the list is provided for informational purposes only and may be revised based on changes in the scope, RFP, contract, or other aspects of the CMS project. It is also important to note that this document provides general guidelines that will be followed for the submittal and approval of CMS project deliverables. Refer to the RFP for detailed and specific requirements pertaining to the content, format, and other requirements relative to the CMS project deliverables. An excerpt of these requirements is provided in Appendix A for reference purposes. 2.2 Important Features of the Deliverable Review Process The deliverable review process is presented in the flow charts in Appendix C of this document. The flow charts illustrate the specific steps in the deliverable review process and are followed by a matrix that provides a narrative description of each step. In summary, the process features: Completion of a Deliverable Expectation Document (DED) for each deliverable to help ensure that the State and the contractor possess a mutual understanding of the scope, objectives, content, and format of each deliverable. The contractor must complete the DED prior to commencing work on the deliverable. A template for the DED is provided as an attachment (in a separate file, rather than an Appendix) to this document. Contractor walkthroughs of each deliverable (if requested by the CMS Project Manager) to clarify content, answer questions, or to familiarize the project team with the deliverable. Formal review and written comments on deliverables by key project stakeholders, with the use of a standardized matrix to document comments. Refer to Appendix D for a template of this matrix. CMS Project Page 3 Deliverable Review Process

8 Contractor submittal of draft and final versions of each deliverable to accommodate appropriate revisions, corrections, or other changes requested by the Department. Formal written acceptance of project deliverables by the CMS Project Manager, IT Technical Program Manager, and the CMS Project Sponsor. Archiving of all project deliverables in the CMS project library. 2.3 Identification and Notification of Deliverable Reviewers After the CMS Project Manager approves the DED, the CMS Project Manager will identify the individuals who will review and comment on the deliverable. These individuals may vary depending on the scope and subject matter of each deliverable. They may specifically include: CMS project team members, Executives, managers, and staff within the Division of Construction and the Division of IT, The CMS project Independent Verification and Validation (IV&V), and; Other project stakeholders The CMS Project Manager will develop, maintain, and publish a schedule of contractor project deliverables. The schedule will be disseminated to all potential deliverable reviewers to inform them of the timing of all potential deliverables. Once the DED for a deliverable is complete, the CMS Project Manager will identify the individuals who will review and comment on the draft and final deliverables. The CMS Project Manager will notify the reviewers of the key milestones in the review process for the deliverable, including the date that the draft deliverable is scheduled for delivery, due date for comments, and the due date for a final deliverable. The CMS Project Manager will also disseminate a copy of the DED to the deliverable reviewers so that they are aware of the scope, objectives, and content of the deliverable, and are prepared to comment on it once it is complete. 2.4 Deliverable Review Period While reviewing the DED, the CMS Project Manager will determine the amount of time required to review and comment on the draft and final deliverables. The baseline time period for review and comment will be: CMS Project Page 4 Deliverable Review Process

9 Not less than ten (10) State business days for review and comment on draft deliverables; Not less than five (5) State business days for review and comment on final deliverables; and, The State will have up to 30 business days to review a draft or a final deliverable The actual review and comment period specific for each deliverable will be included in the project schedule. The project schedule timeframes should be used by the contractor to establish the completion dates for draft and final deliverables. These dates should be included in the DED. During the review of the DED, the CMS Project Manager will validate the timeframes for review and determine whether any changes to the due date of the deliverable or the review and comment period are necessary. The contractor must make changes to the date/period that the CMS Project Manager determines. The ultimate goal, however, will be to change the dates preceding the due date of the final deliverable, but not the final deliverable due date in the project schedule. 2.5 Comments on Draft Deliverables Deliverable reviewers will be responsible for documenting their requested changes in a matrix presented in Appendix D. The CMS Project Manager will compile and review comments to draft deliverables prior to forwarding them to the contractor for incorporation into the final draft. During this review process, the CMS Project Manager may consolidate, revise, or eliminate comments as necessary. The CMS Project Manager will be responsible for communicating any of these changes to the reviewer who initially made the comment. 2.6 Finalizing Deliverables The contractor will be responsible for updating the draft deliverable and producing the final deliverable. The contractor will clearly identify changes to the draft document when the final document is submitted (this may be accomplished by tracking changes in the draft document, although other methods may be appropriate and accepted). In addition, the contractor must use the matrix in Appendix D to document how the comment was addressed in the final version of the deliverable. 2.7 Deliverable Acceptance Criteria The RFP identifies general acceptance criteria for deliverables. Refer to Appendix A for a description of these criteria. These criteria, in addition to any other agreed upon criteria, should be documented by the contractor in the draft version of the DED. CMS Project Page 5 Deliverable Review Process

10 In addition to the criteria in the RFP, the CMS Project Manager may identify additional criteria specific to each individual deliverable. The CMS Project Manager may direct the contractor to add these criteria to the final version of the DED. In addition to the acceptance criteria in the RFP, examples of acceptance criteria may include: Does the deliverable address all components required by the contract and any areas agreed upon subsequently through meetings and planning sessions (these agreements should be reflected in the DED)? Is the deliverable consistent with the content and format agreed upon in the DED? Does the deliverable meet applicable industry and State of California standards? Does the deliverable meet the project and the Department s business and technical needs? Is the deliverable easy to understand and effectively formatted? 2.8 Formal Deliverable Acceptance Deliverables will not be considered approved by the Department until they have been accepted by the CMS Project Manager. Acceptance will be documented in writing on a notice of deliverable acceptance. The CMS Project Manager will send the contractor a copy of this form as a written notice of deliverable acceptance. Unless otherwise stated in writing by the CMS Project Manager, the following individuals have the authority to formally accept a deliverable: CMS Project Sponsor IT Program Manager CMS Project Manager The deliverable will not be considered accepted unless the notice of deliverable acceptance has been signed by each of these individuals. Payment for contractor deliverables will not be made unless the deliverable is formally approved. It is important to distinguish between the roles of deliverable reviewers and deliverable approvers as shown in the flowcharts in Appendix C. Deliverable reviewers are project team members, subject matter experts, stakeholders, and managers and staff within the Division of Construction or the Division of IT who will be asked to review and comment on deliverables for quality control purposes. Their feedback may or may not CMS Project Page 6 Deliverable Review Process

11 be forwarded to the contractor for incorporation into the final version of a contractor deliverable, subject to the discretion of the deliverable approvers. Deliverable approvers have the ultimate authority to approve and formally accept a deliverable. In this capacity, they must sign the notice of deliverable acceptance as indicated above. They may also provide comments on contractor deliverables as part of the deliverable review process. CMS Project Page 7 Deliverable Review Process

12 3 NEXT STEPS This process will be implemented once the contractor has been selected and commences work. As a first step to implementing this process, the CMS Project Manager will work with the contractor s project manager and the project team to ensure that the milestone due dates for all deliverables including DED s, draft and final versions, and the review and comment period are clearly identified in the project schedule. After project schedule is complete, the CMS Project Manager will develop the list of contractor deliverables and their due dates, and disseminate the list to key stakeholders and the potential deliverable reviewers. The process described in this document should be implemented when the contractor begins developing the deliverables. CMS Project Page 8 Deliverable Review Process

13 APPENDIX A RFP CONTRACTOR DELIVERABLE REQUIREMENTS

14 CMS PROJECT Contactor Deliverable Requirements The contractor deliverable requirements below are extracted from RFP DOT , as of October 23, It is important to note that all contractor deliverable requirements have been included in this Appendix for reference purposes only. The contractor must adhere to the requirements applicable to the submittal of document deliverables as part of the process included in this document. Development and Implementation Deliverable Minimum Content Requirements Reqmt # CD-1a CD-1b CD-1b (contd) Deliverable Title The Project Management Plan must adhere to the Office of the State Chief Information Officer California Project Management Methodology described in the Statewide Information Management Manual, Section 17 and the Department of Transportation Information Technology Project Management Office s standards and requirements. The Project Management Plan must include: 1- Scope Management Plan, including how scope issues will be resolved 2- Human Resources Plan, including: a) Organizational charts b) Depiction and description of authority and reporting structure c) Staff roles and responsibilities d) Approach to managing contractor and subcontractor staff e) Coordination of staff at multiple, remote locations including meeting and work status coordination f) How resource shortages will be resolved g) Contractor s management structure and its methods for oversight and executive direction for the contract 3- Communications Plan that describes how the Contractor team will interact and communicate with State personnel including the Project Manager, project team, stakeholders, and internal and external department oversight entities 4- Schedule Management Plan that discusses how schedule delays will be resolved 5- Quality Management Plan 6- Risk Management Plan 7- Issue Management Plan 8- Conflict Resolution Plan that includes an escalation process for resolving issues and ensuring adequate executive direction and oversight 9- Document Management Plan including a) Method and location for document storage b) Standards for document format and content c) Approach to preparing documents and deliverables d) Approach to assigning and controlling document versions CMS Project Page A-1 Deliverable Review Process

15 Reqmt # CD-1b (contd) Deliverable Title 10-Product Development Lifecycle Plan (PDLC), describing the configuration / development approach (e.g., RUP, waterfall), including a) System Design Phase 1-Definition and review of system, software and hardware detailed requirements 2-Implementation of a Change Control Process 3-Development of physical, logical, interface and server diagrams 4-Development of a data dictionary 5-Business process mapping to technical requirements b) System Development Phase 1-Site preparation activities 2-Data conversion activities 3-Coding / configuration activities 4-Interface development activities 5-Report development activities c) System Testing Phase 1-Code and unit testing 2-Integration testing 3-System integration testing 4-Performance and load / stress testing 5-Field testing 6- Regression testing 7-User acceptance testing d) System Training Phase 1-Administrator training 2-Train-the-Trainer training 3-Project staff training e) System Implementation Phase 1-System roll-out 2-Adminsitrative documentation, including disaster recovery procedures f) Production Support and Transition Lifecycle 1-Help desk services 2-Problem fixes 3-Change / enhancements process 4-Backups and disaster recovery CMS Project Page A-2 Deliverable Review Process

16 Reqmt # CD-2a CD-2b CD-2c CD-2d CD-3 CD-4 CD-5 Deliverable Title 5-Knowledge transfer 6-Issue resolution g) Contract Closeout Lifecycle 1-Engagement completion 2-Documented outstanding contractual deficiencies The Project Schedule must conform to the Department s high-level phase Work Breakdown Structure: Initiation, Planning, Execution, and Closeout. The Project Schedule must include the contractor s proposed Project Development Lifecycle (PDLC) which is detailed in the Execution Phase. The Project Schedule must include: 1- Work Breakdown Structure 2- Key Milestones 3- Deliverables 4- Detailed Tasks 5- Task Dependencies 6- Durations for each Task 7- Planned and Actual Start and End Dates 8- Resources for each Task 9- Task Ownership, including Contract and Caltrans Tasks 10- Percent of Project Complete 11 Critical Path The Project Schedule shall contain tasks that are, at a minimum, defined such that the lowest level tasks are no greater than twenty (20) calendar days in duration. The Contractor shall use the Caltrans Weekly PMO Status Report template for creation of the Biweekly Status Report. The template is available from the Bidders Library. The Steering Committee Status Reports / Presentations must specify: 1- Current progress against milestones 2- Coordination needs, such as participation from the districts and / or other divisions 3- Potential risks or opportunities and mitigation / contingency plans 4- Pros / Cons and analyses of key issues These status reports and presentations will be delivered when requested by the State. The System Development / Configuration Plan must specify: 1- Description of the Software Development Methodology and Processes, including Development / configuration standards and tools to be used for: --Information modeling --Networking and transport protocols --Security and encryption --Directory access --Web services protocol stack Programming languages to be used (if applicable) Development / configuration, test and support tools to be used Approach to using and configuring third-party tools and software Approach to reusable software (if applicable) CMS Project Page A-3 Deliverable Review Process

17 Reqmt # CD-5 (contd) CD-6a CD-6b Deliverable Title Approach to validating and refining the functional requirements Approach to managing requirements and traceability Design, code and testing standards Approach to design, code and test walkthroughs / reviews Approach to development and testing of CMS interfaces Approach to designing and validating system security throughout the Software Development Life Cycle (SDLC) 2- Approach to Testing and QA for the CMS Approach to product and process reviews Approach to deliverable quality reviews Approach to testing and test management Approach to integration and testing of interfaces and third-party software Approach to managing defects and anomalies, including how the fixes will be incorporated back into the main code line 3- Approach to Development / Configuration and Deployment of the Phased Releases Approach to configuration management of the system environments and software (Configuration Management Plan) Change Control Management Plan Approach to version and release management How the system will be audited and turned over to the State upon final system acceptance Approach to delivering and managing the phased releases Approach and sequencing of the releases (i.e., what order, how overlapping releases are addressed, how coordinated with users and stakeholders) Approach to verifying readiness to proceed to the next release Approach to training for each release Approach to deploying each release Approach to field test for Function 3 (daily diaries and mobile devices) Risks to the phased release deployment based on the proposed solution and approach The System Requirements Specification must contain: Summary Of The Results Of The Requirements Analysis And Validation Baseline Requirements Set For The System, including any additional derived requirements and expanded definitions of each base requirement in order to demonstrate Contractor understanding Listing Of New Business Processes Required And / Or Impacts To State Policies An Updated Requirements Traceability Matrix, showing traceability from the RFP requirements to the detailed requirements which refine and further elaborate the requirement and associated business rules. The functional requirements in Exhibit VI-A, Mandatory Functional and Technical Requirements, and Exhibit VI-B, Mandatory Optional and Desirable Optional Requirements CMS Project Page A-4 Deliverable Review Process

18 Reqmt # CD-7 CD-8 Deliverable Title (if the State exercises any of the M/O or Desirable options) shall serve as the baseline requirements that the Contractor shall use as the basis for validating and refining the requirements after contract execution. Modifications to the baseline requirement set are to be expected and will be processed through the System Change Control Process and approved by the State. The System Fit / Gap Analysis must specify: Gaps between the Contractor s proposed solution and the CMS functional and technical requirements Strategies to address the gaps between the solution and the CMS requirements (e.g., configuration, customization, additional products) Impacts and risks of the proposed strategies Identification of State responsibilities related to addressing the gaps (e.g., policy changes) The High-Level Design must contain: Description of proposed hardware, for each environment including storage Description of proposed software and the purpose of each item Physical Architecture, including --Location of the environments (if hosted solution) --Description of the facilities (if hosted solution) --System architecture for each environment (i.e., development, test, training and production environments) --Proposed network design, including firewalls --Physical and logical system diagrams Physical and logical integration of the products (technical view) including --Approach to software integration, as applicable --Logical integration diagram or technical diagram showing proposed software integration --List of all required software integration points, including which software, versions, etc. will be integrated and for what purpose --Technical description of each software integration point, including discussing how the integration is proposed to be done --Description of data groups that will be shared through integration points --Approach to integration with mobile devices Application architecture (functional / user view) --Approach to navigation and user interface --Approach to reports and ad hoc queries --Approach to forms and correspondence Description of proposed system security features --User account management for internal and external users (e.g., types of roles, managing roles and access to contracts) CMS Project Page A-5 Deliverable Review Process

19 Reqmt # CD-8 (contd) CD-9 CD-10 Deliverable Title --Security for transmission and uploading of files Approach to interfaces Approach to capacity management --Performance modeling calculations and assumptions --Calculations used to size the CMS hardware and solution --Approach to addressing growth during and after the contract period --Approach to estimating, verifying and managing system performance --Archive strategy for management of data Approach to meeting system performance and system availability requirements The Detailed Design Document and Database Schema must contain: Capabilities, functions and features of the system Description of the overall design Description of software modules, interfaces, forms, correspondence, and reports Software design using narrative, pseudo code, tables, flow charts, data flow diagrams, screen formats / layout, etc. to describe or depict the inputs, outputs, data flows, control flows, manual and automated processing, and exception and error handling (as applicable to the type of solution) Description and design of the system workflow processes, including decision points and approvals, sequential / parallel routing, exceptions, and relationship to new / modified business processes Description of the system s role-based user access / authority features and tools Report layout for standard reports Form and correspondence layouts Data model and relationships Database(s) design and schema System interface design Specific formulas and calculations used to establish the system storage and processing needs (capacity and performance modeling) Description of the systems security features for protecting the system, data transmissions, and data from unauthorized access, including, as appropriate, encryption, auditing and logging An updated requirements traceability matrix, showing traceability to the specific areas of the design which describe how the requirement will be implemented The Interface Specification Document must specify: Discussion of each interfaces and its purpose Transmission methods and tools The approach to implementing each of the interfaces, including --Integrating with real-time systems (e.g., E-FIS) --Interfacing with stand-alone DBs / batch systems For each interface, a description of: --Interface method and protocol --Frequency of transmission CMS Project Page A-6 Deliverable Review Process

20 Reqmt # CD-11 CD-12 CD-13 Cd-14 CD-15 Deliverable Title --Estimated data volume and growth patterns --Interface format, including any control and error formats --Error handling and recovery methods, including restart procedures --Dependencies or constraints for the interface (e.g., timing of the transmission) --Interface security considerations --Interface testing considerations and approach --Operational considerations, such as scheduling, log file monitoring and operator notifications The Detailed Data Extract / Mappings (for loading Caltrans-converted data) must specify: Database Schema for the new databases Data mapping (legacy data elements to new data elements) Input and output record layouts Location of source and target data File / table sizing information File / table volume information The Test Plan for each testing phase (i.e., unit, integration, system, performance (including load / stress testing), field, regression, and user acceptance) must contain: Overall test strategy Description of each test phase Test objective and test strategy Approach to creating and maintaining the test data and test scripts High-level test design and test cases Approach to testing interfaces, as applicable Criteria for exiting the test phase Test documentation to be produced before, during and at the close of the testing activities Approach to managing, controlling, coordinating and communicating changes to the test environment and code during the test phase Test suspension activities and test resumption activities Approach to analyzing and resolving test anomalies and errors Roles and responsibilities of Contractor and State staff for the testing process The Test Scripts / Procedures for each test must contain: Specific test objectives, including requirements to be verified Specific test preparation activities, including setup of the test data and test environment Specific test activities with expected results and actions to be taken Specific analysis of the test results to verify correctness Specific criteria used to determine if the test has passed or failed The Contractor shall provide all test data for the unit, integration, system, performance, and regression test phases. The State will provide the test data for the field and acceptance tests. The Test Results Report for each testing phase must contain: A summary of all tests executed and the test results A detailed summary of all anomalies and defects found during testing and their resolution or status A list of the system configuration prior to the start of testing and at the end of testing (e.g., versions of software modules, system settings, etc.) CMS Project Page A-7 Deliverable Review Process

21 Reqmt # CD- 16a CD- 16b CD-17 CD-18 CD- 19a Deliverable Title An updated requirements traceability matrix showing traceability to the specific test(s) which prove satisfaction of the requirement. A substantiated recommendation for either resolution of problems (and further testing) or progression to the next phase The Training Plan must specify: Approach to train-the-trainer training to State trainers Approach to assessing user competencies and evaluations Approach to addressing refresher and remedial training Strategy for delivering training to Caltrans internal users and the CMS external users (i.e., contractors) that will be used by the Caltrans trainers Specific course curriculums and descriptions Approach to Computer-Based Training (CBT) for internal and external users (if CBT is optioned) Proposed training schedule Roles and responsibilities of Contractor and State staff for the training process The Contractor must provide all training materials for the Train-the-Trainers training, UAT training, and the System Administration / Technical training. The Train-the-Trainer Training Materials must include: Specific goals of the training class Methods for training end users Description of system features and functions, including workflows Description of new / modified business processes / policies Exercises to help reinforce concepts for end users Training Instructor Manuals for End User Training, for each release must include: Specific goals of the training class Topics and features to be covered in the course(s) How to setup and use the training environment for the course(s) Description, instructions, and answers for the class exercises Key concepts that must be conveyed Tips and tricks for using the system The Internal and External End User Training Materials must include: Course outlines for each end user training session Course content for each end user training session, including as appropriate: Description of each system function / module / feature Description of the data elements and their range of values and business rules Description of system workflows and Day-in-the-Life scenarios Hands-on and workbook exercises to reinforce system concepts Tips and key references for navigating the system Use of standard and ad-hoc reporting tools and features Creating queries and reports Specialized report writing training for advanced users Student materials used during the training course(s) Worksheets or exercises that can be used to evaluate training effectiveness and student proficiency CMS Project Page A-8 Deliverable Review Process

22 Reqmt # CD- 19b CD-20 CD-21 CD-22 CD-23 Deliverable Title Data to support the training exercises Note: Caltrans will be responsible for making copies of the materials for each end user training session. The External End User Training Materials shall be designed as self-paced training materials. If the State elects to purchase the mandatory optional CBT component, this requirement will be deleted in favor of the CBT for External Users. The UAT Training shall consist of the following topics: 1- Overview of the UAT process and purpose (i.e., how UAT will be conducted, roles and responsibilities); 2- Specific UAT management processes, including test preparation, test execution, recording and analyzing individual test results, test closeout, reporting bugs and anomalies, and test status reporting; 3- System basics, including logging in, navigation, workflow, and reporting; and 4- Overview of business processes that are applicable to the Function(s) being tested (e.g., for Function 1, how to setup a contract, how to update the contact list, how to generate a payment estimate). The Knowledge Transfer Training Plan (for technical training) must include: Approach to direct delivery training to Caltrans super-users and technical staff (e.g., job shadowing, classes, etc.) Training delivery methods Specific course curriculums and descriptions Strategy for knowledge transfer to technical staff Proposed knowledge transfer methods Proposed activities where State staff would participate Proposed technical training and knowledge transfer schedule Roles and responsibilities of Contractor and State staff for the training process The Training Materials for System Administrator / Technical Training must describe: Course outlines for each technical training session Course content for each technical training session, including as appropriate: Description of each system environment / component Hands-on exercises to reinforce system concepts Tips and key references for operating, maintaining and troubleshooting the system Use of appropriate administration, monitoring and reporting tools and features Security considerations Student materials used during the training course(s) Worksheets or exercises that can be used to evaluate training effectiveness and student proficiency Data to support the training exercises This plan may not be applicable to a hosted solution, depending on the amount of configuration vs. out-of-the-box functionality. The State shall have the final decision as to whether the plan is applicable. The User Manual(s) must describe: Features and functionality of the system System workflows, including mandatory data elements, expected actions for each workflow step, range of values, specific validations and business rules at each workflow step and tips to assist the users CMS Project Page A-9 Deliverable Review Process

23 Reqmt # CD-24 CD-25 CD- 26a Deliverable Title Procedures for creating ad hoc reports and correspondence Supervisory controls and special features only available to managers and supervisors, including reassignment of their staff s work, handling exceptions, and overriding workflow steps Description of codes values Description of online help features Expanded description of error messages, as appropriate The System Administration Manual(s) must describe specific procedures for: Management of internal and external user accounts and security Error and exception diagnosis and handling, including checklists Changing system constants and parameters (e.g., rates, deadlines, etc.) Reviewing and managing system and database logs for general activities, problems and trends Adding and modifying data elements and user-defined fields Creating and modifying standard reports, correspondence / templates and forms Creating and modifying system workflows Creating and modifying business rules and field validations Archival storage and retrieval Monitoring and managing interfaces and data transmissions Updating online help contents Activities for system monitoring and administration Routine periodic maintenance to be performed The Deployment and Transition Plan must specify: Deployment strategy (i.e., preparation, who, when, how, verification) Proposed criteria for Go / No-Go decision to commence system deployment Approach to loading converted data Proposed detailed deployment schedule Approach to managing the deployment for each system release, including Process for coordination and timing of training Process for establishing / expanding the help desk Process for tracking and resolving anomalies and errors during deployment Process for communicating with users during the deployment Criteria used to ensure the CMS release is functioning correctly Proposed decision points for shutting down the legacy system(s), as applicable Roles and responsibilities of Contractor and State staff during the deployment period Approach to transitioning operational responsibilities to State staff upon system acceptance Specific risks associated with the proposed deployment approach and specific mitigation / contingency plans for the identified risks The System Configuration Report must contain: Description of the hardware and network configurations for each environment The current versions of all COTS and custom software modules, including the date and time stamp of the version Any hardware settings or configurations made to the system production environment The database schema and database configuration information including table descriptions CMS Project Page A-10 Deliverable Review Process

24 Reqmt # CD- 26b CD- 28a Deliverable Title and sizing Any supporting files that must be in place for correct system operations (e.g., environmental files, help files) Note: The purpose of this report is to ensure the State has sufficient information to rebuild and restore the system in the event of catastrophic failure. The level of detail provided in this report may vary depending on the type of solution proposed by the Contractor. The Final System Configuration Report must describe the final state of the production environment when the system is turned over to the State at the end of the CMS Production Acceptance Period. Any and all custom-developed software code created in support of the CMS must be delivered to the State. This code includes all items necessary to build and deploy the application (i.e., modify, enhance, compile and deploy), including but not limited to: Source code and code libraries, including appropriate comments and source code headers Software utilities, build scripts, or tools developed Configuration and environmental files (e.g.,.dll,.ini, etc.) Reports / queries, correspondence / templates, and style sheets CD- The Contractor must provide documentation for all custom-developed software code 28b including any Contractor-developed pre-defined queries and reports. Cd-28c The software documentation for any custom-developed software must include: Description of the security and user access / authority features Purpose, capabilities, functions, and features of the code Description of any dependencies or relationships with other modules, code or interfaces Charts and accompanying descriptions depicting inputs, outputs, data flows, control flows, and manual and automated processes CD- All software development and testing tools must be off-the-shelf. Open source and freeware 29a is acceptable for development and testing tools only. CD- The Contractor must provide all of the software tools necessary to develop, build, test, and 29b deploy the CMS, including automated testing tools. CD- The Contractor must provide the off-the-shelf development / configuration tools necessary to design, develop, and implement the CMS. 29c CD- 29d CD-30 CD-31 CD-32 CD-33 The Contractor must deliver at least one (1) copy of any and all development / configuration, testing, training, configuration, system monitoring, and maintenance / operations tools used in support of CMS with licenses to support five (5) concurrent users for each tool provided. The Contractor must provide an off-the-shelf automated testing tool(s) for CMS to perform regression, performance and load / stress testing. The Contractor must provide off-the-shelf system monitoring tools for CMS. The Contractor must provide COTS tools necessary for the maintenance and operation of the system. If third-party COTS software is being proposed as part of the production systems (i.e., if the software is necessary for the operation of the production system), the Contractor must provide software licenses to support 2,300 named internal users and 1,700 named external users. CMS Project Page A-11 Deliverable Review Process

25 APPENDIX B LIST OF DELIVERABLES SUBJECT TO THE DELIVERABLE REVIEW PROCESS

26 CMS PROJECT List of Contractor Deliverables Subject to the Deliverable Review Process This process contained in this document applies to the submittal and review of document deliverables by the contractor. A list of document deliverables, extracted from RFP DOT , as of October 23, 2009, is presented below. Deliverable # CD-1 CD-2 Contract Deliverable List and Due Dates Deliverable Title Update Due Date Frequency Project Management Plan, including Project Schedule Updates to the Project Schedule / Work Plan At Least Quarterly Every two (2) weeks Thirty (30) State Business Days after Contract Execution Thirty (30) State Business Days after Contract Execution Associated Services A.4.2 A.4.2 CD-5 System Development / Configuration Plan CD-6 System Requirements Specification As needed As needed, based on approved change requests Sixty (60) State Business Days after Contract Execution Six (6) months after Contract Execution A.4.2 A.4.3 CD-7 System Fit / Gap Analysis No updates Eight (8) months after Contract Execution CD-8 High-Level Design Every release In accordance with the State-approved Project Schedule CD-9 CD-10 CD-11 Detailed Design Document with Database Schema Interface Specifications Document Detailed Data Extract Mappings (for loading Caltrans-converted data) Every release As needed, based on approved change requests As needed In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule A.4.3 A.4.3 A.4.3 A.4.3 A.4.4 CMS Project Page B-1 Deliverable Review Process

27 Deliverable # CD-12 Deliverable Title Test Plan for each phase of testing: a) Unit (if custom code is required) Update Frequency Every release Due Date In accordance with the State-approved Project Schedule Associated Services A.4.5 b) Integration (if custom code is required) c) System d) Field e) Performance f) Regression g) User Acceptance CD-13 Test Scripts / Procedures for each phase of testing: a) Unit (if custom code is required) Every release In accordance with the State-approved Project Schedule A.4.5 b) Integration (if custom code is required) c) System d) Field e) Performance f) Regression g) User Acceptance CD-14 Test Data for each phase of testing: a) Unit (if custom code is required) Every release In accordance with the State-approved Project Schedule A.4.5 b) Integration (if custom code is required) c) System d) Field e) Performance CMS Project Page B-2 Deliverable Review Process

28 Deliverable # Deliverable Title Update Frequency Due Date Associated Services f) Regression g) User Acceptance CD-15 Test Results Report for each phase of testing: a) Unit (if custom code is required) Every release In accordance with the State-approved Project Schedule A.4.5 b) Integration (if custom code is required) c) System d) Field e) Performance f) Regression g) User Acceptance CD-16 Training Plan Every release In accordance with the State-approved Project Schedule CD-17 CD-18 CD-19 CD-20 CD-21 CD-22 Train-the-Trainer Training Materials Training Instructor Manuals for End User Training Training Materials for Internal and External End Users Training Materials for UAT Training Knowledge Transfer Training Plan (for technical training) Training Materials for System Administrator / Technical Training Every release Every release Every release Every release As needed Every release In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule A.4.6 A.4.6. A.4.6. A.4.6. A.4.6. A.4.6. A.4.6. CMS Project Page B-3 Deliverable Review Process

29 Deliverable # Deliverable Title Update Frequency Due Date CD-23 User Manual(s) Every release In accordance with the State-approved Project Schedule CD-24 CD-25 CD-26 CD-27 CD-28 CD-29 CD-30 CD-31 CD-32 System Administration Manual Deployment and Transition Plan System Configuration Report Coding Standards, if applicable to the solution Custom Developed Software Code and Documentation, if applicable to the solution Software Development Tools, including software, manuals, manufacturer warranties and licenses Software Testing Tools, including software, manuals, manufacturer warranties and licenses System Monitoring Tools, including software, manuals, manufacturer warranties and licenses Software Maintenance and Operations Tools, including software, manuals, manufacturer warranties and licenses, if applicable to the solution Every release Every release Every release and at the end of the CMS Production Acceptance Period In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule No updates At least thirty (30) calendar days prior to the start of coding activities Every release No updates No updates No updates No updates Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Five (5) State Business Days after System Acceptance Associated Services A.4.6 A.4.6 A.4.7 A.4.7 A.4.3 A.4.3 N/A N/A N/A N/A CMS Project Page B-4 Deliverable Review Process

30 Deliverable # CD-33 CD-34 CD-35 Deliverable Title Third-Party Software required for system operation, including software, manuals, manufacturer warranties and licenses, if applicable to the solution Computer-Based Training (CBT) for Internal Users (Mandatory Optional) Computer-Based Training (CBT) for External Users (Mandatory Optional) Update Frequency No updates Every release Every release Due Date Five (5) State Business Days after System Acceptance In accordance with the State-approved Project Schedule In accordance with the State-approved Project Schedule Associated Services N/A N/A N/A CMS Project Page B-5 Deliverable Review Process

31 APPENDIX C DELIVERABLE REVIEW PROCESS FLOW CHARTS

32 CMS PROJECT Deliverable Review Process Flow Charts The pages which follow present flow charts depicting the CMS deliverable review process. The flow charts are followed by a matrix that presents the narrative description of each task depicted on the flow chart. While following this process, the contractor must adhere to the RFP Contractor Deliverable Requirements presented in Appendix A of this document. Should any inconsistencies or conflicts exist between this document and either the RFP or contract, amendments, or agreements, the RFP or contract, amendments, or agreements will take precedence. CMS Project Page C-1 Deliverable Review Process

33 CMS PROJECT DELIVERABLE REVIEW PROCESS Flowchart #1: Submittal, Review, and Approval of Deliverable Expectation Document CMS Project Manager Contractor Start (1) Review Deliverable Review Process Document (3) Document Receipt Of Draft DED, Disseminate for Review (2) Complete and Submit Draft Deliverable Expectation Document (DED) (5) Compile Comments Matrices and Review Comments (6) Finalize and Submit Comments Matrix to Contractor (7) Finalize DED and Submit for Approval (8) Corrections Made As Agreed? YES (10) Approve / Sign Off On DED NO (9) Determine Remediation Steps (11) Send DED to Deliverable Reviewers (12) Archive DED in Project Library To Flowchart #2 Deliverable Reviewers (4) Review DED and Complete Comments Matrix Deliverable Approvers CMS Project Page C-2 Deliverable Review Process

34 CMS PROJECT DELIVERABLE REVIEW PROCESS Flowchart #2: Submittal and Review of Draft Deliverables Contractor From Flowchart #1 (13) Complete and Submit Draft Deliverable (15) Conduct Deliverable Walkthrough (20) Conduct Walkthrough of Comments Matrix To Flowchart #3 CMS Project Manager (14) Document Receipt of Deliverable (16) Distribute Deliverable for Review (18) Compile Comments Matrices and Review Comments (19) Finalize and Submit Comments Matrix to Contractor Deliverable Reviewers Deliverable Approvers (17) Review Deliverable and Complete Comments Matrix CMS Project Page C-3 Deliverable Review Process

35 CMS PROJECT DELIVERABLE REVIEW PROCESS Flowchart #3: Submittal, Review, and Approval of Final Deliverables CMS Project Manager Contractor From Flowchart #2 (21) Finalize and Submit Deliverable (22) Document Receipt Of Deliverable (23) Conduct Deliverable Walkthrough (24) Corrections Made as Agreed? (25) Determine Remediation Steps (26) Submit to Deliverable Approvers for Acceptance / Signature (28) Archive and Disseminate Final Deliverable End Process Deliverable Reviewers Deliverable Approvers (27) Approve / Sign Off On Deliverable CMS Project Page C-4 Deliverable Review Process