Assessment of NCTD Program Management Framework for Positive Train Control (PTC) Program. Subtask 1: Discovery Current State Revision 3

Transcription

1 DELIVERABLE Assessment of NCTD Program Management Framework for Positive Train Control (PTC) Program Subtask 1: Discovery Current State Revision 3 Contract Number: OS Task Order 1 April 2013 Submitted to: Brad Hansen, M.S., PMP North County Transit District Director, PMO Capital Projects 810 Mission Avenue Oceanside, CA 92054

2

3 Table of Contents Table of Contents 1. Introduction Discovery Subtask Key Observations Integration Management Program Governance Program Control Change Management Program/Project Management Information Systems Scope Management Program Requirements Scope Definition Work Breakdown Structure (WBS) Schedule Management Schedule Development Critical Path Cost Management Capital Program Costs and Budget On-Going Operating and Capital Costs (Post-Implementation) Quality Management Quality Plan PTC Safety Plan Human Resource Management Program Organization Roles and Responsibilities Risk Management i icfi.com/transportation

4 Table of Contents 8.1 Risk Management Plan Risk Prioritization Risk Registers Procurement Management Approval Authorities Vendor List Stakeholder Management Communications Management PTC Outreach Plan Project-Specific Communication Plans Meetings and Reports Next Steps Appendix A: List of NCTD Personnel Interviewed... A-1 Appendix B: PTC Program Retreat Agenda... A-3 Appendix C: PTC Program Charter... A-7 Appendix D: NCTD Organizational Chart... A-9 Table of Exhibits Exhibit 1: NCTD PTC Project Construction Organization... 4 Exhibit 2: PTC Project/Contract-Level Change Management Process... 5 Exhibit 3: PTC System Contractor Critical Dates Exhibit 4: PTC Program Budget (Updated January 2013) Exhibit 5: Current versus Baseline Budget Allocations Exhibit 6: Contractor Reporting Relationships (Quality Plan) Exhibit 7: Project Manager Assignments Exhibit 8: Risk Probability Matrix Exhibit 9 - NCTD's PTC-Related Risk Registries ii icfi.com/transportation

5 Table of Contents Exhibit 10: PTC Stakeholders Exhibit 11: Stakeholder Engagement Strategies iii icfi.com/transportation

6 Table of Contents 2013 iv icfi.com/transportation

7 Introduction 1. Introduction The Rail Safety Improvement Act of 2008 (RSIA) mandated the implementation of Positive Train Control (PTC) systems by intercity passenger and commuter rail operators. As a result, the North County Transit District (NCTD) commenced the acquisition and implementation of PTC technologies in August 2011 and has already executed several contracts. Inspections of NCTD s PTC implementation by the Federal Railroad Administration (FRA) are scheduled to commence in August, 2014 in advance of the federally mandated date for full implementation of December 31, The IBI Group (IBI) and ICF International (ICF) are assisting NCTD in evaluating its program management (PM) framework and identifying critical PM needs for the PTC Program. The objective of this assessment is to review existing processes, tools, and artifacts; assess any gaps; and make recommendations on the staff resources, processes, tools, and artifacts required to strengthen project planning, execution, monitoring and control to enable the expeditious, cost-effective delivery of NCTD s PTC Program. The assessment is being accomplished through the execution of three sequential subtasks; Discovery, Gap Analysis, and Plan of Action. The purpose of the Discovery subtask is to establish a factual baseline of the current state of the PTC Program what exists, what does not exist, and what has occurred to date at NCTD. This forms the foundation of the Gap Analysis, where the factual baseline of NCTD s current state of the program will be compared against the Project Management Institute (PMI) Project Management Body of Knowledge (PMBOK) practices and other federal and non-federal program/project management guidance and best practices and gaps identified. Finally, the Plan of Action provides a set of recommended actions to address the gaps and strengthen the program to successfully deliver the PTC Program and its component projects.this report summarizes the results of the Discovery subtask only. 1.1 Discovery Subtask The objective of the Discovery subtask is to identify and understand the current state of the NCTD PTC Program relative to the following key program and project aspects, as identified in the PMI PMBOK Integration Management Scope Management Schedule Management Cost and Budget Management Quality Management Human Resource Management Procurement Management Stakeholder Management Communication Management icfi.com/transportation

8 Introduction Under the Discovery subtask, ICF interviewed key PTC Program stakeholders (refer to Appendix A for a list of interviewees); reviewed current NCTD Program artifacts provided by staff; and observed a two-day PTC Program retreat (refer to Appendix B for the retreat agenda) attended by representatives of NCTD s senior management. The subtask was accomplished over a ten-day period, five days of which were at NCTD s facilities in Oceanside, California. 1.2 Key Observations Following are some key initial observations drawn from the Discovery process. PTC is a complex and dynamic project where much is unknown, yet NCTD is working under an aggressive, Federally-mandated deadline. There is evidence of project-level management discipline at NCTD, including the use of processes, templates, and reporting standards. At the program-level, there was little evidence of formally documented program controls that integrated scope, schedule, and budget for all PTC-related activities for either vendorimplemented or NCTD-implemented projects. Approximately 40% of the total program budget is allocated to one contractor (HTI) causing much of the program s focus to be on the activities that fall within that contractor s scope of work. While individual contracts and certain activities have schedules associated with their particular project element of the PTC Program, there is no evidence of a integrated master schedule that encompasses and pulls together every program and project element. It is not clear that there has been sufficient rigor in the PTC Program cost estimation process and whether or not the remaining PTC Program contingency will be sufficient to address project risks, required future scope changes, and under-budgeting of program elements that are unknown at this time. There does not appear to be a documented (post-implementation) operations and maintenance and capital cost estimate associated with the PTC system. There does not appear to be an integrated configuration management process at the programlevel that addresses pre and post-implementation PTC-related configuration considerations. There does not appear to be a sufficient level of understanding of and engagement in the PTC Program among project team members and key NCTD staff and how they (can or should) affect the direction of the PTC Program and/or how they are (or will be) affected by the program. These and other initial observations will be further explored and developed in the subsequent Gap Analysis and Plan of Action icfi.com/transportation

9 Integration Management 2. Integration Management Integration management includes the processes and activities needed to identify, define, combine, unify and coordinate the various proceses and project management activities associated with disparate, but related, work elements and projects within a program. In this case, integration management focuses on all the projects and supporting activities, to include both vendor-implemented and internally-implemented, necessary for the success of the PTC Program. There are several components needed for effective integration management, including (but not limited to) Program Governance, Program Controls, Change Mangement Processes, and Program/Project Management Information Systems. The current NCTD PTC Program is described below in the context of these elements. 2.1 Program Governance Program governance refers to the decision-making, execution, and accountability structures, plans, processes needed to advance the program. A Project Management Plan (PMP) forms the foundation of that governance and provides critical direction for the entire program and its component projects. NCTD developed and issued a formal PTC PMP on January 4, 2012 and recently made an initial revision on March 22, The current version of the PMP contains the major elements typically found in a PMP for a program of this size and complexity. The Project Charter (refer to Appendix C) is incorporated in the PMP and communicates, in bulleted format, a series of goals and priorities to be accomplished throughout the execution of the program. The governance structure used to oversee and administer the program is presented in Exhibit 1, dated March 18, 2013, and was provided by NCTD in response to a document request as part of the Discovery process icfi.com/transportation

10 Integration Management Exhibit 1: NCTD PTC Project Construction Organization Section 7 of this report (Human Resource Management) provides a more in-depth description of the PTC Program organization. 2.2 Program Control In the months leading up to this assessment, NCTD s Director of PMO/Capital Projects, Mr. Brad Hansen, implemented a series of project-level controls. Out of 15 active vendorimplemented projects within the PTC Program, 13 have implemented these project-level controls. The controls include: standardized folder and file structures on a shared drive used by PTC personnel formal change management and communications plans and associated templates standardized project workbook that captures project-specific decisions, issues, risks, and change requests formal templates for weekly status reporting and change requests stored project schedules At the program-level, there was little evidence of formally documented program controls that integrated scope, schedule, and budget for all PTC-related activities for either vendor-implemented or NCTD-implemented projects icfi.com/transportation

11 Integration Management 2.3 Change Management The ability to effectively govern and manage changes to the program and its component projects is critical. Part 8 of the PMP (Change Management, Claims, Disputes and Conflict Resolution) addresses some aspects of change management, though it reads to be largely contract-based change management. At the project-level, a formal change management process was recently implemented (refer to Exhibit 2 below). These changes refer specifically to the contracts and their respective scopes as opposed to the governance of program level changes to scope, schedule, and/or budget. Exhibit 2: PTC Project/Contract-Level Change Management Process Recognize Change Submit Change Request (PCR Form) Log Change Request Origin & disposition of changes processed through the PCR process Evaluation Evaluate PCR & Recommend Action Decision Approved, rejected, parked (record in Change Log & Submit REQ if applicable Integration Integrate change into the Project Documents Communication Communicate change to team The process depicted in this Exhibit does not match the process outlined in Part 8 (Change Management) of the PMP. Additionally, based on the materials reviewed by the assessment, a documented integrated change control process at the program-level was not discovered. 2.4 Program/Project Management Information Systems Systems that provide timely, accurate, and relevant information to the program manager, project managers, other project team members, and internal and external stakeholders are essential to effectively managing scope, schedule, budget, and other ancillary functions. The PTC Program is using e-builder Enterprise to facilitate capital improvement program evaluations, issue PTCrelated Request for Proposals (RFP), process change orders, and track costs, among other functions. E-Builder is known as a construction program management software package. In an e-builder press release, dated February 22, 2012, NCTD s PTC Program Manager, Mr. Jay Walser, is quoted regarding NCTD s implementation of e-builder as stating, We have a large number of funding sources we need to track. Our goal is to end up with a centralized and consistent repository of information across the agency, one we can use for institutional knowledge throughout a project s lifecycle, from initial concept to close-out. There is no reference to e-builder in the latest version of the PMP. Additionally, Part 10 (Resource Management) of the PMP states that NCTD will establish a Microsoft SharePoint site to manage and disseminate a variety of program artifacts to include, but not limited to, system plans, communications plans, schedules, and operating practices. To date, this site has not been implemented. Lastly, RMI s RailDOCS product is being implemented to collect and manage signal configuration data, equipment test reports, and other data. The latest Project Status Report, dated icfi.com/transportation

12 Integration Management March 22, 2013, indicates that the system will be fully implemented by June 30, 2016 six months after the Federally-mandated deadline of December 15, 2015 for full PTC system functionality icfi.com/transportation

13 Scope Management 3. Scope Management Scope management refers to the tools and processes used to define and control the entire body of work required to implement a service, product, or project. A clearly defined, comprehensive scope that addresses all aspects of the program and its component projects is essential. The foundation of a well-crafted scope is a clear understanding of the requirements of program and its component projects what the program must accomplish and how it must function. Defining all the work required to complete the program successfully is critical it is just as important to have clarity as to what is not included as part of the program s scope. Just as the program scope must be proactively managed throughout the life of the project, so must the program requirements to ensure that all program and project deliverables are able to effectively meet the defined requirements through appropriate testing and verification. A critical outgrowth of a well-defined, comprehensive program scope is a Work Breakdown Structure (WBS). The WBS takes the program scope and its component project elements and breaks it down into detailed tasks and subtasks that must be accomplished. A thorough, well crafted WBS is a key building block for the sequencing of all project activities, estimation of project resources, and development of a master schedule discussed in subsequent sections of this report. 3.1 Program Requirements A clear definition of the PTC Program requirements is embodied in federal regulation. Specific guidance is provided in 49 Code of Federal Regulation (CFR) Requirements for Positive Train Control systems which outlines the specific requirements that fully compliant PTC systems must meet. Part 5 (Requirements Management) of the PMP offers an overview of the PTC Program s approach to defining and verifying PTC system requirements. Of note, the PTC Program defines its approach to requirements management as to assure the PTC Systems Contractor defines documents, verifies, and meets the needs and expectations of its customers, internal and external stakeholders, and FRA regulations. Additionally, the PMP states that, HTI will be responsible for the traceability of requirements on the NCTD PTC Program. Herzog Technologies, Inc. (HTI) is a Prime contractor to NCTD for the development and implementation of the PTC system. HTI s contract value represents roughly 40% of NCTD s total budget for the PTC Program. While HTI has contractual responsibilities in support of PTC development and implementation, NCTD as the system owner is ultimately responsible for meeting all program requirements. Until recently implemented project-level controls, contractor requirements were not being tracked in a systematic manner. No evidence of coordinated program-level requirements management was observed. 3.2 Scope Definition Guidance pertaining to the full scope of PTC systems can be found in 49 CFR 236, Subpart I - Positive Train Control Systems. Part 4 (Scope Management) of the PMP offers a description of NCTD s approach to scope management with respect to the PTC Program. The description refers to an original Project Scope statement, though the location of this original scope statement was not found. Additionally, a Scope Change Request Form is identified and a icfi.com/transportation

14 Scope Management process for its completion and submission is described. However, the location of this form and what entity receives the form submitted is unknown. In response to a request for a documented definition of the program s scope (work breakdown structure and associated dictionary), the assessment team was provided with a document entitled NCTD Detailed Work Plan, dated February 6, The detailed work plan provides good context for the PTC system development, implementation, and commissioning of an FRAcompliant system. However, elements of the plan are redundant with the current version of the PMP, revealing inconsistencies between the two plans. 3.3 Work Breakdown Structure (WBS) A comprehensive, detailed WBS was not located within the NCTD Detailed Work Plan. Though, a WBS and associated data dictionary were found during review of NCTD s Risk Management Plan for the PTC Program, dated October 12, The WBS is organized by program phases (i.e. initiation, planning, execution, control, and closeout) and appears to have inconsistencies with its accompanying data dictionary icfi.com/transportation

15 Schedule Management 4. Schedule Management Schedule management refers to the discipline of identifying and monitoring all the tasks, milestones, deliverables, and associated interdependencies, required for successfully delivering a service or product within a time context. At the program-level, an integrated master schedule is a project management best practice and an essential planning, management, and communications tool throughout the implementation of the program and its various component projects. The master schedule clearly identifies the critical path of the program and its component projects. The critical path is the sequence of schedule activities that determines the duration of the project. The critical path determines the minimum total project duration. In schedule development it is critical to accurately define the critical path and ascertain which project tasks are dependent upon the completion of a previous element and which tasks can proceed in parallel until a defined point. Proactive schedule management ensures that program and project team members can work in concert to minimize delays and develop work-arounds when feasible. 4.1 Schedule Development The project charter includes a reference to the development of an Integrated Master Program Schedule. While individual contracts and certain activities have schedules associated with their particular project element of the PTC Program, there is no evidence of a master integrated schedule that encompasses and pulls together every program and project element. Part 15 (Schedule Management) of the PMP offers an overview of NCTD s approach to schedule management. In its approach, the PTC Program states that it is employing three-levels of schedules with varying degrees of complexity: Level 1 Executive Summary: a one page schedule summary in Gantt chart format showing design, bid and award, procurement/construction, systems installation, testing and start-up and the major milestones for each major area of the Program. Level 2 Summary Program Schedule: a Gantt chart format with one or more bars per contract, showing the typical activities of design, bid and award, procurement, construction, and systems installation. Level 3 Detailed Project Schedule: a fully integrated Critical Path Method (CPM) schedule presented in a resource loaded Gantt chart format. Based on its review of program artifacts, it appears that NCTD is employing some, but not all, of the levels described above. Project-level schedules are stored for seven of the 15 identified vendor-implemented PTC projects; they included: HTI PTC updated September 2012 North Fiber updated September 2012 South Fiber updated November 2012 HTSI CTC updated February 2013 Macro CTC same as HTSI CTC RailDOCS updated March icfi.com/transportation

16 Schedule Management In addition to these project-level schedules, several Integrated Master Schedules were provided; they included: HTI Project Schedule PTC Construction PTC Rollup Although these schedules are identified as integrated master schedules, a single, comprehensive, primary schedule depicting the sequencing, interdependencies, and timing of all the major tasks, milestones, and deliverables of the program was not observed. 4.2 Critical Path Lastly, within Part 11 (Project Controls) of the PMP, NCTD highlights defined critical dates for the PTC System Contractor within the schedule filed with the approved PTCIP [PTC Implementation Plan]. Exhibit 3 below summarizes these dates. Exhibit 3: PTC System Contractor Critical Dates Milestone Publish PTC System Requirements for Proposals PTC System Bids Due Contractor Selection Contractor Notice to Proceed Completion Date Complete Complete Complete Complete PTC Design and Construction First Quarter 2014 PTC Construction Complete First Quarter 2014 Completion of System Safety Certification and Safety Plans Submittals Second Quarter 2014 PTC in Revenue Service December 31, 2015 It is important to note that PTC System Contractor refers to HTI. While these critical dates and milestones are important from a contractual point of view, there does not appear to be a clear, comprehensive array of all program-wide subordinate and ancillary milestones that must be met in order to successfully deliver the program and a definition of the critical path (and durations) through which key milestones must be accomplished icfi.com/transportation

17 Cost Management 5. Cost Management Cost management refers to the processes involved with estimating, budgeting and controling costs so that the program can be completed within the approved budget in this case for the PTC Program. 5.1 Capital Program Costs and Budget Part 14 (Financial Management) of the PMP provides an overview of NCTD s cost control and reporting processes for the PTC Program. The PTC is currently estimated to cost approximately $87.3 million and is currently funded through nine different funding sources, including Federal, state, and local resources. Exhibit 4 below presents how the current PTC Program budget is allocated across nine cost categories. The PTC system and the communications infrastructure (CBI supporting the transmission of data for the PTC system) represent 57% of the overall program budget. Exhibit 4: PTC Program Budget (Updated January 2013) Dispatch System, $2,731,501, 3% Consultants, $4,289,471, 5% Construction Support, $4,679,688, 5% Wayside Signal Upgrades, $6,550,000, 8% PTC, $35,746,088, 41% CBI, $13,739,723, 16% Labor & Caltran Fee, $7,252,900, 8% OCC, $6,082,148, 7% Contingency, $6,221,450, 7% Exhibit 5 below presents how the current budget allocation differs, by cost category, from the original budget submission in July Initial cost projections for CBI and wayside unit upgrades were lower than ultimately required. As a result, the PTC Program was required to reallocate nearly 70% of its contingency fund and reduce the scope of the operations control center. It is not clear that there has been sufficient rigor in the PTC Program cost estimation process and whether or not the remaining PTC Program contingency will be sufficient to address project risks, required future scope changes, and other under-budgeting of program elements that are unknown at this time icfi.com/transportation

18 Cost Management Exhibit 5: Current versus Baseline Budget Allocations Description Original FRA Budget Current Budget Net Changes increase / (decrease) PTC 35,793,738 35,746,088 (47,650) Contingency 19,360,697 6,221,450 (13,139,247) Operations Control Center (OCC) 10,900,000 6,082,148 (4,817,852) Labor & Caltrans Fee 6,712,950 7,252, ,950 Communication Backbone Infrastructure (CBI) 4,750,000 13,739,723 8,989,723 Construction Support 3,290,124 4,679,688 1,389,564 Consultants 3,000,000 4,289,471 1,289,471 Dispatch System 2,835,460 2,731,501 (103,959) Wayside Signal Upgrades 650,000 6,550,000 5,900,000 TOTALS 87,292,969 87,292,969 - PTC Program and project costs are tracked by NCTD s Finance division using five job codes in JD Edwards, NCTD s cost control system. The PTC Program is also tracking program estimates and associated costs using e-builder. Instead of tracking costs across the five job codes used by the Finance division, the PTC Program is categorizing its estimates and costs across the 15 identified vendor-implemented projects. To date, these projects represent a total expenditure of $52.2 million with $21.2 reserved for future project capacity. The PTC Program finance representative is currently working with the NCTD Finance division to reconcile the cost accounting differences between e-builder and JD Edwards. 5.2 On-Going Operating and Capital Costs (Post-Implementation) There do not appear to be rigorous, documented (post-implementation) operations and maintenance and capital cost estimates associated with the PTC system once it is implemented. PTC Program representatives state that there s inherent difficulty in forecasting future costs associated with the PTC system given that much of the technology is new and as the system and its subsystems mature, costs are further refined. As stated in the PMP, due to the relative state of development of the PTC technology NCTD s Original PTC Project Budget was developed without the benefit of much of the detailed information and assumptions required to produce a project budget with a high degree of confidence or precision. Current estimates of recurring costs range from $1.5 million to $3 million annually based on conversations with PTC Program personnel during the discovery process these estimates appear to be anecdotal and not analytically based icfi.com/transportation

19 Quality Management 6. Quality Management Quality management refers to the processes and activities that ensure the program s requirements, are met and validated in this case for the PTC Program. It implements the quality management system through policy and procedures. 6.1 Quality Plan Part 9 (Quality Management) of the PMP states that a Quality Process will be defined by HTI and will augment NCTD s Quality Plan and fulfill requirements set forth in 49 CFR 235, subparts H and I. Specifically, the PMP outlines the requirements by which HTI s quality process must conform, to include the following areas: Records management Audit procedures Personnel training and qualification Document control Configuration management The PMP also identifies the use of a closed loop reporting system, Failure Reporting, Analysis, and Corrective Action System (FRACAS), to be used by HTI and its subcontractors throughout the execution of the PTC project. As a part of FRACAS, the PMP states that a Failure Review Board (FRB) will be established to review failure trends and ensure that corrective actions are taken for identified failures. No documentation or other evidence of the FRB s existence was observed. The PTC Program s Quality Plan, dated January 1, 2012, provides an overall quality assurance process and quality-related responsibilities of contractors supporting the PTC Program. Specific topics included in the Quality Plan are: Quality organization Quality system Design control Configuration control Document control Quality records Procurement control Inspection and test control Audit procedures Deficient item control Within its Quality Plan, NCTD depicts the contractor reporting relationships, presented in Exhibit 6. However, based on the data collected during the assessment team s initial site-visit and document review a quality management function at the organizational level is not evident,. Therfore, it is unclear how the reporting structure depicted in Exhibit 6 relates to NCTD s current organization-wide structure, Also of note, Parts 7 and 12 (Quality Records of the Quality Plan) of the PMP are duplicative. Lastly, there does not appear to be an integrated configuration management process at the program-level that addresses pre and post-implementation configuration considerations icfi.com/transportation

20 Quality Management For example, processes for managing changes to the configuration of PTC-related equipment needs to be established. Such equipement would include (but not be limited to) items that are located within the right of way, on-board the locomotives, and in the back office. Lack of sufficient configuration control could have far reaching adverse impacts on future rail system operations, maintenance, and safety. Exhibit 6: Contractor Reporting Relationships (Quality Plan) 6.2 PTC Safety Plan A PTC Safety Plan (PTCSP) must be submitted to the Federal Railroad Administration (FRA) and receive a PTC System Certification before placing the system into service. Federal regulation 49 CFR lists the content requirements of the PTCSP, one of which states that the PTCSP must contain a complete description of the safety assessment and Verification and Validation processes applied to the PTC system, including their results. Naturally, the description of verification and validation processes are good sources of information with respect to the discipline of quality management. NCTD s Division Chief for Safety gave a presentation on safety-related aspects of the PTC Program. Based upon the information delivered in the presentation, Rail Sciences Corporation (RSC) is developing NCTD s PTCSP. The final PTCSP will be reviewed by NCTD s Safety Division before submission to the FRA, which must occur at least 120-days before FRA testing of NCTD s PTC system is to commence. It should be noted that a file entitled PTC Safety Plan (PTCSP) was provided to the assessment team in response to a document request. Upon review of the PTCSP provided, it appears that the file does not represent the formal PTCSP to be submitted to the FRA, but a Project Site-Specific Safety Plan developed by HTI and submitted to NCTD. A description of the safety assessment and Verification and Validation processes to be applied to the PTC system, as required by 49 CFR , could not be located in this version of the PTCSP icfi.com/transportation

21 Human Resource Management 7. Human Resource Management Human resource management refers to the staffing of resources necessary to deliver and manage the program and its component projects in this case the PTC Program Team. Its foundation includes a program organizational structure and clearly defined roles, responsibilities, and accountabilities among program and component project team members 7.1 Program Organization Section 2.1 (Program Governance) of this report contains and organizational chart of NCTD s PTC Program. This chart is presented again below for ease of reference. The PTC Program organization, referred to as the PTC Project Construction Oversight Organization, is depicted in the organizational chart above. It is a projectized organization with slight characteristics typically associated with a matrixed organization. A projectized organization is defined as one in which the program manager has authority to assign priorities, apply resources, and direct the work of individuals assigned to the project. With respect to the PTC Program, the Chief Rail Systems Officer (also referred to as the PTC Project Manager, PTC Program Manager, PTC Program Director in various program artifacts) directs the day-to-day activities of the personnel assigned to the program. The current structure and role definition of the Chief Rail Systems Officer appears to be largely geared for the oversight of contractors responsible for implementing elements of the PTC system throughout NCTD s operational environments icfi.com/transportation

22 Human Resource Management The Chief Rail Systems Officer reports directly to the NCTD Chief Executive Officer (CEO) along with NCTD s seven Division Chiefs (refer to Appendix D for the NCTD-wide organizational chart). Part 18 (Appendix A Organization Structure) of the PMP also presents an organizational chart for the program, referred to as the PTC Program Management Organizational Structure. Although this organizational chart is included in the most recent version of the PMP, it does not match the organizational chart depicted in Exhibit 1. To date, NCTD has identified 15 vendor-implemented projects within the PTC Program, 14 of which are currently active. Five Project Managers have been identified and assigned to oversee specific vendor-implemented projects (organized by contract). Exhibit 7 identifies these Project Managers and the projects they are assigned to oversee. Exhibit 7: Project Manager Assignments Name Formal Title Projects Vendor Luis Bedoya Elisa Castro- Pacheco Roy Gilbert Robert Paladino Eric Roe Project Manager 1. North Coaster Fiber Install 2. South Coaster Fiber Install 3. San Diego Coaster Fiber Install 4. Environmental Support 5. DSDC & Final Design fiber & BNSF JCT Final Design Project Manager 6. PTC System 6. HTI Senior Network Engineer Rail Systems Engineer Deputy Chief Rail Systems Officer 7. PTC Network Design 8. TMDS Network Extension 9. Centralized Traffic Control (CTC) 10. CTC Pjt Mgt & OCC Prelim Design 11. RailDOCS Configuration Mgt SW 12. Test & Training Facility (TTF) Equip 13. CTC System Design 14. WIU Design &Installation 15. Design Support During Construction (DSDC) 1. Fishel Company 2. MP Nexlevel of CA 3. TBD (future) 4. BRG Consulting 5. HNTB 7. Datel Systems 8. Datel Systems 9. HTSI 10. Macro 11. RailCar Management 12. Datel Systems 13. PRE 14. PRE 15. PRE 7.2 Roles and Responsibilities Part 3 (Program Management Organizational Responsibilities) of the PMP includes a narrative description of the PTC organization. The description contains key responsibilities and accountabilities, related to the PTC Program for the CEO, PTC Program Director, Deputy PTC Program Director, and PTC Project Managers. Also included in this Part of the PMP is a listing of the Participating NCTD Divisions that provide indirect support to the program, including: Finance Accounting Grants Contracts and Procurement icfi.com/transportation

23 Human Resource Management Quality Human Resources Information Technology Network Design Network Support Operations Safety Development Services Civil Engineering Facilities Management Interagency Coordination and Communications Chief of Staff Community Outreach Media Communications Public Relations In addition to the staff responsibilities outlined in the PMP, NCTD provided documented roles and responsibilities for the Rail Systems division (the phrase Rail Systems is equivalent to the PTC Project, as stated by NCTD personnel), dated March 26, 2013; this was provided in response to a document request as part of this Discovery process. The Rail Systems role and responsibilities document provides high-level position descriptions for the following functional areas: Program/Project Management Signaling Communications (rail and signal data) Accounting Office Administration Document Control Contracts The relationship between the aforementioned roles and responsibilities document provided by NCTD and the responsibilities and accountabilities of key personnel documented in Part 3 of the PMP is unclear and inconsistent. Additionally, formally documented skills, experience, and qualifications requirements for each role within the PTC Program were not evident. As a result, it is difficult for the PTC Program to monitor and assess whether it has appropriately equipped itself with the required level and combination of human resources to successfully implement the Program and its individual components icfi.com/transportation

24 Human Resource Management icfi.com/transportation

25 Risk Management 8. Risk Management Risk management refers to the processes by which the programs and component projects/managers proactively identify, analyze, monitor, and, where necessary, mitigate risks. Key aspects of effective risk management include risk management plans, risk prioritization, and use of a risk register. Part 6 (Risk Management) of the PMP provides a brief description of NCTD s approach to risk throughout the PTC Program s lifecycle. 8.1 Risk Management Plan In addition to Part 6 of the PMP, NCTD provided the assessment team with a Risk Management Plan for the PTC Program, dated October 10, There is no cross-reference between either the PMP or the Risk Management Plan making it unclear how Part 6 (Risk Management) in the PMP relates to the Risk Management Plan. The Risk Management Plan offers a more detailed description than the PMP of NCTD s approach to risk management, including narrative descriptions for: Risk identification Risk qualification and prioritization Risk Monitoring Risk mitigation and avoidance 8.2 Risk Prioritization The PMP includes a risk matrix to be used as a quantitative tool to prioritize identified risks based on their probability of occurrence and their impact on the program. Using the matrix, risks are scored 1 through 9; 1 through 3 are categorized as low risk, 4 through 6 as medium risk, and 7 through 9 as high risk. Exhibit 8 depicts the risk matrix included in the PMP. Please note, a level 6 risk is being interpreted as a medium risk despite being titled as High Risk in the matrix below. There is not evidence of the Risk Probably Matrix, depicted in Exhibit 8, currently being used as a risk prioritization tool. Exhibit 8: Risk Probability Matrix From a qualitative perspective, recent project-level controls were implemented that prioritize risks based on a qualitative analyses, discussed further in the next section icfi.com/transportation

26 Risk Management 8.3 Risk Registers A risk register is an effective, dynamic tool that is used to continuously identify, monitor, and manage risks to scope, schedule, budget, and other internal and external concerns thoughout the duration of the program. Part 19 (Appendix B NCTD PTC Risk Assessment Matrix) in the PMP contains a risk register populated with 24 initial risks identified for the program. In addition to the PMP risk register, additional risk registers were identified throughout the Discovery subtask; these included a PTC Program Risk Registry; a listing of top three risks to the program in the Risk Management Plan; and individual project-level risk registries for 12 of the 15 identified vendor-implemented projects. Exhibit 9 below provides a summary of the various risk registers reviewed by the assessment team. Exhibit 9 - NCTD's PTC-Related Risk Registries Title Location Description Part 19 Appendix B NCTD PTC Risk Assessment Matrix Top Three Risks PTC Program Risk Registry Individual Project-Level Risk Registries PMP Risk Management Plan NCTD Share Drive - N:\Public\Divisions\PTC\PTC - Program Risk Rollup Respective project workbooks Risks are depicted in table-format and prioritized based on a function of their probability of occurrence and financial impact to the program. Risks are presented in narrative format and describe a series of environmental and archeological risks to the program; prioritization appears to be largely qualitative in nature. Risks are depicted in table-format and organized by constraint-type and program phase. Prioritization is based on a qualitative rating of High, Medium, and Low this risk register is the most current program-level risk register. Embedded in a project s workbook is a risk register for that particular project. Similar in format to the PTC Program Risk Registry, these registries prioritize risks based on a qualitative analyses. The risk registries are populated to varying degrees, some of which do not list any risks, and do not feed the PTC Program Risk Registry. There are inconsistencies between the multiple risk registers and risk management processes outlined in the PMP and Risk Management Plan icfi.com/transportation

27 Procurement Management 9. Procurement Management Procurement management refers to the processes necessary to obtain the necessary products and services required for success of the PTC Program. Part 16 (Procurement Management) of the PMP outlines NCTD s approach to obtaining products and services related to the PTC Program. The approach defines: Individuals authorized to make purchasing decisions Types of contracts to be used The contract approval process Decision criteria Vendor management Two critical elements of procurement management are Approval Authorities and Vendor Lists; each will be discussed below. 9.1 Approval Authorities The PMP identifies the Chief Rail Systems Officer, Deputy Chief Rail Systems Officer, and the Senior Contracts Administrator as individuals who are authorized to approve purchases for the PTC Program. Additionally, the assessment team was provided with NCTD s Operating and Capital Budget Procurement Signature Authority for Requisitions, Check Requests, Invoices, and Contracts, dated February 14, In that document, the Program Director PTC/CTC is provided with approval authorities for requisitions of up to and including $100,000, and invoices of up to and including $999,999; PTC Project Managers are provided approval limits of $25,000 and $25,000 for requisitions and invoices, respectively. There s no reference in the PMP to the roles and responsibilities of the Senior Contracts Administrator. 9.2 Vendor List Federal regulation 49 CFR (a) requires each railroad implementing a PTC system on its property to manage a PTC Product Vendor List (PTCPVL) that includes all vendors and suppliers of each PTC system, subsystem, component, and associated product, and process in use system-wide. Based on a PTC Vendor List provided to the assessment team on March 29, 2013, the PTC Program has 51 vendors, to date. There appear to be inconsistencies between the vendors listed on NCTD s PTC Vendor List, Part 3.4. (Contractors) of the PMP, and those that are known to be actively supporting the program. As well, there are vendors listed on list that the were not referenced in other vendorrelated documentation reviewed. It is important to note that the provided vendor list is in alphabetical order; the last vendor listed begins with an L suggesting that a subset of vendors were inadvertently cut off from the icfi.com/transportation

28 Procurement Management document provided. Additionally, some of the vendors listed on the PTCVL appear to be internal resources (i.e. individuals) within NCTD icfi.com/transportation

29 Stakeholder Management 10. Stakeholder Management Stakeholder management refers to the processes associated with the identification, analysis, and constructive engagement with the individuals and organizations that are impacted by, or have an impact on, a program in this case, the PTC Program. Stakeholders can be internal or external to the NCTD organization. Developing and executing effective stakeholder strategies is critical to proactively engaging and managing the expectations of; providing timely and accurate information to; and, where appropriate, obtaining input and feedback from each stakeholder group. Effective and proactive internal and external stakeholder engagement must be an integral part of the program scope (including the WBS), schedule (including the critical path), and budget (including in-house and external financial resources) not just an add-on. Part 12 (Communications and Outreach Plan) of the PMP does identify three stakeholder groups; communities, media outlets, and government entities. However, NCTD s current version of the PMP does not include a section devoted specifically to stakeholder management that thoroughly documents 1) the results of an analysis that identifies internal and external PTC Program stakeholder groups and 2) the engagement strategy that will be pursued by the PTC Program for each identified stakeholder group. The NCTD Government Affairs and Communications Division, and their focus on stakeholder engagement and communications outreach, is a critical resource that can greatly support the PTC Program. During the two-day executive retreat devoted to the PTC Program, the communications team delivered a presentation that provided an overview of public outreach considerations with respect to PTC. In their presentation, the communications team identified a series of stakeholders and categorized them accordingly. Exhibit 10 depicts the PTC stakeholder identified by the communications team. Exhibit 10: PTC Stakeholders icfi.com/transportation

30 Stakeholder Management The communications team also provided a high-level approach, or engagement strategy, for each of the stakeholder groups. Exhibit 11 presents these strategies. In addition to the information presented at the two-day retreat, NCTD s communications team also provided a draft Outreach Plan. Within that plan is a section devoted to Targeted Stakeholders and preliminary criteria by which PTC stakeholders will be identified, including: Self-identification (those who have made their interest known) Proximity (those who live near a proposed construction) Mandate (agencies that have some authority or groups that have related missions) Use (people who may gain or lose some use) Economics (those who gain or lose something economically in the process) In addition to these preliminary criteria, the draft Outreach Plan includes a detailed listing of stakeholders organized by region. Exhibit 11: Stakeholder Engagement Strategies icfi.com/transportation

31 Communications Management 11. Communications Management Communications management refers to the processes associated with developing and delivering timely, relevant, and transparent program information to internal and external stakeholders through effective communication channels in this case for the PTC Program and its component projects. Part 12 (Communications and Outreach Plan) of the PMP outlines NCTD s approach to PTC-related communications with contractors, other Program team entities, and the communities through which it operates. The approach outlined in the PMP is largely focused on external stakeholders and the use of NCTD s Government Affairs and Communications Division for managing communications. Key aspects include an outreach plan, project-specific communication plans, and meetings and reports PTC Outreach Plan Supplementing the PMP is a draft Outreach Plan for the PTC Program developed by NCTD s Government Affairs and Communications Division. The Outreach Plan provides a more detailed approach to PTC-related communications to include general public information, interagency communications, and internal management of communications. In addition, a draft communications schedule and budget is included in the Outreach Plan, though the budgetary information is largely unknown at this point. Lastly, a series of communications mechanisms are included in the Outreach Plan and how they can be used to disseminate PTC-related information. These include: Print media Video media Social media Online media Toll free hotline To date, the communications team has developed a PTC-specific website ( and developed a series of press releases and fact sheets. Cross-referencing between the PMP and PTC Outreach Plan was not evident, making it unclear as to how the two documents relate to one another Project-Specific Communication Plans Project-specific communication plans have been developed by each of the project managers for their respective projects. The Communication Plan template requires project managers to identify project-specific stakeholders and internal and external communication processes. The communication plan template also offers an overview of the project structure, though it is unclear how this structure relates to the one described in Part 3 (Program Management Organizational Responsibilities) in the PMP. It appears there are inconsistencies in the terminology used to describe the various roles within the project management structure icfi.com/transportation

32 Communications Management 11.3 Meetings and Reports Through interviews, observations, and artifact reviews, it is apparent that the PTC Program engages in multiple formal and informal meetings and reporting activities. However, a comprehensive listing of these communication mechanisms was not observed in one location. Based on the information collected by the assessment team, the PTC Program either manages or participates in the following meetings for the purpose of communicating and gathering PTCrelated information: Quarterly interoperability meetings with tenants of NCTD s rail Monthly Capital Projects Steering Committee meetings Bi-Weekly Internal stakeholder meetings Weekly Division Chief Meetings Weekly Project Meetings held individually by respective Project Managers and their vendor teams Additionally, based on the information gathered by the assessment team the PTC Program manages the following reporting activity: Quarterly program-level issues, risks, and change reports to multiple funding and oversight organizations Yearly programmatic updates to multiple funding and oversight Despite the multiple meetings and reports described above, there does not appear to be a sufficient level of understanding of and engagement in the PTC Program among project team members and key NCTD staff and how they (can or should) affect the direction of the PTC Program and/or how they are (or will be) affected by the program icfi.com/transportation

33 Next Steps 12. Next Steps This Discovery report was intended to provide a snapshot of the current state of the PTC Program Based on the information gathered from this Discovery effort, the assessment team will next perform an analysis to identify gaps in the NCTD program and project management processes, artifacts, tools and staff resources required for effective management of the PTC Program. During the next subtask, particular attention will be focused on NCTD s PMP. The purpose of the PMP is to enhance PTC Program planning and implementation efforts and provide essential, consistent guidance to program team leaders and members both in-house and contractors. As part of its gap analysis, ICF will review NCTD s draft PTC PMP and other PTC-related plans and practices to ensure consistency with 1) best practices outlined in PMI s Project Management Body of Knowledge (PMBOK); 2) ICF s experience and lessons learned from analogous programs, tailored to the NCTD PTC Program; 3) PMP content requirements outlined in Title 49 of the Code of Federal Regulations, Part 633 Project Management Oversight, Subpart C Project Management Plans Contents of a project management plan; 4) the FRAmandated schedule for PTC implementation nationwide; and 5) relevant Federal Transit Administration (FTA) guidance to grantees. In addition, NCTD identified the following components of the PMP as priority-areas during ICF s review: Program management organization the organizational structure and associated resources employed by NCTD to execute, monitor and control the overall implementation of the PTC System Requirements management plan the mechanism NCTD employs to document and reconcile and implementation of PTC System requirements as contract deliverables throughout the program s lifecycle Risk management plan the process employed by NCTD for identifying, forecasting, and prioritizing programmatic (e.g. scope, schedule, and cost) and technical (e.g., requirements and maintenance) risks throughout the program s lifecycle The results of the gap analysis will be delivered in a subsequent report to the NCTD Director, PMO Capital Projects for appropriate distribution. Building upon the gap analysis, the assessment team will then develop a Plan of Action that identifies recommendations to address gaps and strengthen NCTD s capacity to successfully deliver the PTC Program icfi.com/transportation

34 Next Steps icfi.com/transportation

35 Appendices Appendix A: List of NCTD Personnel Interviewed The following individuals were formally interviewed as part of the Discovery subtask: Matthew Tucker, Executive Director Brad Hansen Director, PMO Capital Projects Jay Walser Chief Rail Systems Officer David Murphy, Chief Operations Officer Ryan Cashin, Chief Technology Officer Ryan Bailey, Chief Financial Officer Reed Caldwell, Chief Development Sevices Officer Bridget Hennessey, Chief of Staff/Intergovernmental Affairs Officer 2013 A-1 icfi.com/transportation

36 Appendices 2013 A-2 icfi.com/transportation

37 Appendices Appendix B: PTC Program Retreat Agenda Day 1 8:30 8:45 a.m. Attendee Check-in (March 27, 2013) Sheraton Carlsbad Resort & Spa 5480 Grand Pacific Drive, Carlsbad, CA PRESENTERS 8:45 a.m. 12:00 p.m. Understanding of all activities related to PTC (including project briefings from all major contractors) JW - Introduction: 20min JH - PTC Prime: 60min o PPT & Schedule o ** JW input (PPT) JW - Flagging/Sig Maintainer: 10min o PPT & Resource Allocation JW CTC: 5min JW - Fiber: 15min o New drawing JW - Design & Oversight: 30min o PPT & Contractors List RC - Network: 20min o PPT ** Includes 2, 10-min Breaks o Introduction Jay Walser o PTC Prime Project (PTC, Test & Training Facility, & Hy-Rail Test Vehicle work streams) Jim Hanlon HTI Jay Walser o Flagging & Signal Maintainer Requirements Jay Walser o Centralized Traffic Control (CTC) Jay Walser o Fiber Jay Walser o Design & Oversight Contractors Jay Walser 12:00 p.m. 1:30 p.m. Lunch If time allows, review Action items/questions from Parking Lot/other o Network Ryan Cashin 2013 A-3 icfi.com/transportation

38 Appendices 1:30 p.m. 2:30 p.m. 2:30 p.m. 2:45 p.m. 2:45 p.m. 3:30 p.m. 3:30 p.m. 4:00 p.m. Review and Update Project Schedule(s) RC Network Schedule: 20min JW Community Outreach Schedule: 15min JW - PTC Program Rollup Schedule: 25min o Note: communication tool (present Div. Chief & CPSC meetings) o PMP status Break Identify Risks (challenges) New Consolidated Risk Log o Pulls from each PM Project Workbook (new), other - Review of Action Items from Parking Lot/other - New Business Open Forum End of Day 1 o Network Schedule Ryan Cashin o Community Outreach Schedule (includes fiber) Jay Walser o PTC Program Roll- Up Schedule + Status of Plans Jay Walser o Jay Walser o Team/Elisa Castro Day 2 8:30 8:45 a.m. Attendee Check-in (March 28, 2013) Sheraton Carlsbad Resort & Spa 5480 Grand Pacific Drive, Carlsbad, CA PRESENTERS 8:45 a.m. 9:30 a.m. 9:30 a.m. 10:30 a.m. 10:40 p.m. 11:30 p.m. Complete Compliance and Financial Review Review & Update Project Budget Review & Update Public Outreach Plan and Conducting Public Outreach ** Includes a Break (10min) Review & Update Safety Plan Provide PTC Status: o PMP o Implementation Plan (FRA Requirement) o Development Plan (FRA Requirement) o Safety Plan (FRA o Ryan Bailey o Deb Castillo o Tom Tulley 2013 A-4 icfi.com/transportation

39 Appendices 1:00 p.m. 2:00 p.m. 2:00 p.m. 2:15 p.m. 2:15 p.m. 3:00 p.m. 3:00 p.m. 4:00 p.m. Review and Update Interagency Coordination Plan Break Developing and publishing updated Comprehensive Project Status Reports and Plans New Project Manager Folder/File Structure N Division PTC-CTC PTC Project Reporting Folders o New Project Plans (e.g. Communication Plan, Change Management Plan) o Contractor SOW-Agreement o New Weekly Meeting Minutes- Action Items (uniform template) o New Project Manager Project Workbook (Projects by PM; uniform template) o Decision-Issue-Risk- Change Request Logs, SOW Deliverables- Requirements Log, Communications Plan Appendix A o New Project Status Reporting (PSR) Process o Individual Project Status which includes Community Outreach (uniform template) o Consolidated High Level Project Rollup Status (uniform template) Public Divisions PTC o PTC Finance Project Reports (JDE Summary & PTC PSR Weekly Rollup) o PTC Project Risk Rollup o PTC-NCTD Stakeholder Meeting Minutes-Action Items - Review of Action Items from Parking Lot/other - New Business Open Forum o Eric Roe o Eric Roe o Team/Elisa Castro 2013 A-5 icfi.com/transportation

40 Appendices 2013 A-6 icfi.com/transportation

41 Appendices Appendix C: PTC Program Charter 2013 A-7 icfi.com/transportation

42 Appendices 2013 A-8 icfi.com/transportation

43 Appendices Appendix D: NCTD Organizational Chart 2013 A-9 icfi.com/transportation