Internal Audit Report. Toll Operations Contract Management TxDOT Office of Internal Audit

Transcription

1 Internal Audit Report Toll Operations Contract Management TxDOT Office of Internal Audit

2 Objective To determine whether the Toll Operations Division (TOD) contract management structure is designed and operating to drive effective vendor performance. This audit covered contract management activity and oversight to examine business activities related to the TxTag billing and customer service processes from June 28, 2013 (contract acceptance) to February 28, Opinion Based on the audit scope areas reviewed, control mechanisms are unreliable in addressing risk factors and exposures considered significant relative to impacting reporting reliability and operational execution. The organization's system of internal controls does not provide reasonable assurance that key goals and objectives will be achieved. Control gap corrections are pervasive and resulting in material negative impacts to the organization. Progress can be made within two quarters in addressing the highest risk areas. Overall Engagement Assessment Unsatisfactory Title Findings Control Design Operating Effectiveness Finding 1 Contract Management/Oversight Framework x x Unsatisfactory Finding 2 Performance Measure Achievement x Unsatisfactory Management concurs with the above findings as identified during the audit test period noted. Management has prepared management action plans and begun to take action to address deficiencies. Control Environment Toll Operations Division (TOD) is responsible for developing, operating, and maintaining TxDOT-owned toll facilities in Austin, Laredo, and the Houston region. TOD currently operates with a staff of 17 full-time equivalent employees, with assistance from a vendor who provides system design, marketing, web support, systems integration, call center/customer service center operations, violation identification, collections, and court representation. TOD also relies on a separate third-party vendor to provide consulting, operational support, integration coordination and subject matter expertise of toll operations. TOD and these two vendors began a conversion process in July 2013 (i.e., notice to proceed) to transition from a previous vendor's toll operations system to a more modern customer service center system. Delays were experienced throughout the conversion process through the go-live date of July 9, 2014 due to issues with the migration, conversion, and implementation of tolling data from the legacy system. These delays and issues drove further issues with system reporting and customer statement releases. Additionally, the vendor had not completed specific performance related tasks which have delayed important contractual milestones from being achieved timely. Effective May 11, 2015, a new Director of the Toll Operations Division was hired to fill the role vacated through retirement. Rating August

3 Summary Results Finding Scope Area Evidence TOD s contract management and oversight activities during the tolling system conversion/implementation noted the following: Documentation did not exist to effectively evaluate achievement of: o system implementation o Customer Service Center (CSC) operations, o established transaction processing goals Documentation could not be provided to define the Quality Control/Quality Assurance process used to validate system data and performance indicators Limited documentation demonstrating monitoring of third-party vendor performance in integration coordination and support of toll operations Billing/Customer Service 1 2 Contract Performance Actions/Damages Contract Performance Actions/Damages Limited documentation to support the following key decisions: o approval for extension to complete contractual milestones o vendor completion of deliverables per milestone o approval to process customer billings Evaluation of milestone completion for system implementation noted the following: Operations Commencement (Milestone 7): Required system testing was not completed as scheduled for the site acceptance, system integration, and parallel operations testing, resulting in delayed implementation of system go-live Retention of the previous toll operations vendor resulted in that vendor receiving additional fees and having damages forgiven to ensure continued operations due to data conversion delays Final Acceptance (Milestone 8): System Final Acceptance scheduled for June 2014 had not occurred resulting in additional delays in the ability to monitor vendor performance Reports requested for ten performance indicators tested were either incomplete or calculations could not be reperformed as data from the system was deemed unreliable or incomplete. Audit Scope The audit included the assessment of contractual liquidated damages related to vendor achievement of milestones including data migration, commencement of operations, and August

4 operational readiness. There were eight milestones that were to be completed by the golive date of July 9, 2014, but as of this report date, two milestones have not been completed. Finally, an audit evaluation was performed of the TOD oversight and monitoring of vendor performance through 1) contract and invoice review, 2) inquiry of key management personnel from TOD and vendors, and 3) other documentation review for the same audit test period. The audit was performed by Tracey Garza, Krista Gronniger, Jaime Resendez, Jennifer Stanush, Kathy Baca, and Bill Hunter (Engagement Lead). Audit fieldwork was conducted during the period from February 6, 2015 to March 24, 2015 to examine business activities conducted from June 28, 2013 (contract acceptance) to February 28, Methodology The methodology used to complete the objectives of this audit included: Vendor contract reviews Review of TOD and vendor plans for data migration and system implementation Testing of specific identified issues for accuracy and reliability of reporting, including: o Selected a judgmental sample of TxTag accounts with pay by mail billing activity and reviewed to determine how statement issues were identified and the corrective action taken o Selected a judgmental sample of merged toll customer accounts and reviewed the process for merging these accounts and validation of accounts to determine accuracy and completeness of conversion o Reviewed planned billing cycles compared to the statement distribution log to identify variances in billing dates o Selected a judgmental sample of toll accounts with duplicate transactions and reviewed to determine how issues were identified and the corrective action taken Reviewed vendor processes for testing interoperability between toll entities Determined contract-related milestones and performance measures Reviewed vendor self-reported performance and evaluated liquidated damages, to date Tested vendor invoices for payments and verified TOD s assessment of damages Reviewed documentation to support monitoring and verification of vendor performance Reviewed communications between TOD and the vendors to determine level of oversight Background This report is prepared for the Texas Transportation Commission and for the Administration and Management of TxDOT. The report presents the results of the audit of Toll Operations Contract Management which was conducted as part of the Fiscal Year 2015 Audit Plan. All toll road revenue has grown from $77 million in FY 2011 to $211 million in FY The current vendor contract was bid using the FY 2011 revenue numbers. August

5 In conjunction with its third-party consultant, TOD issued a Request for Proposal and entered into an agreement with a vendor to modernize the customer service center system in June This modernization included converting existing toll accounts to a new system, merging accounts to simplify account maintenance for the customer, implementing a new interactive voice response system, and updating customer service capabilities to improve the customer experience. Table 1 below provides a timeline of key events related to the conversion. Table 1 Conversion Timeline: Key Events (May 2013-November 2014) *Milestones 6 and 8 are not completed as of the report date Source: Contract documentation provided by Toll Operations Division Toll operations customer billing is divided into two primary types of transactions: 1) tag holders (including private and business TxTag accounts with interoperability with other toll systems across the state) and 2) pay-by-mail customers (those without a TxTag). Private tag holders can prepay a deposit on their accounts and have the ability to maintain their accounts online. As the deposit is expended, the tag holder can opt to make additional payments or setup an automatic replenishment option using a credit card set up on the account. Account activity is charged at the higher pay-by-mail rate for customers whose account balances reach zero and do not make either additional payments or opt for the automatic replenishment option. We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards and in conformance with the International Standards for the Professional Practice of Internal Auditing. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. Recommendations to mitigate risks identified were provided to management during the engagement to assist in the formulation of the management action plans included in this report. The Office of Internal Audit uses the Committee of Sponsoring Organizations of the Treadway Commission (COSO) Internal Control Integrated Framework version A defined set of control objectives was utilized to focus on reporting and operational goals for the identified scope areas. Our audit opinion is an assessment of the health of the overall control environment based on (1) the effectiveness of the enterprise risk management activities throughout the audit period and (2) the degree to which the defined control objectives were being met. Our audit opinion is not a guarantee against reporting misstatement and reliability or operational sub-optimization particularly in areas not included in the scope of this audit. August

6 Detailed Findings and Management Action Plans (MAP) Finding No. 1: Contract Management/Oversight Framework Condition Toll Operations Division (TOD) contract management and oversight activities during the conversion to the new customer service center system were not fully designed, documented, and executed to provide reasonable assurance that 1) the implementation of the customer service center system was completed timely, and 2) implementation, billing, and operational issues identified during the conversion process were prioritized and addressed to achieve effective resolution. Additionally, there was no independent assessment by TOD of vendor performance measures to determine whether customer service and billing activities were performed or computed in accordance with contract requirements or proper control techniques. Effect/Potential Impact The new customer service center system implementation experienced delays in data migration, system testing, operations commencement, and achievement of project acceptance. Since commencement of operations on July 9, 2014, customer service and billing issues were experienced. A complete assessment of vendor contract performance and any resulting damages that might be incurred based on that performance could not be properly determined for the two outstanding milestones and on-going customer service performance measures. Criteria Industry guidance such as The State of Texas Contract Management Guide provides guidelines on performance monitoring and administration: Monitoring contractor performance ensures performance is in accordance with the contract and allows the managing division to be aware of and address any developing problems or issues Development of performance measure deliverables, which clearly outline how results will be measured, are the sole responsibility of the managing division Maintaining a complete contract administration file provides a source to settle claims and disputes that may arise and should contain items such as: o copies of all general correspondence related to the contract o records/minutes of all meetings including sign-in sheets and/or agendas o copies of all routine reports required by the contract such as approval requests and inspection reports Other industry standards, such as the PMI Project Management Body of Knowledge (PMBOK Guide) Fifth Edition, also provide the following guidance: Project governance provides a comprehensive, consistent method of controlling the project, and ensuring success by defining, documenting, and communicating reliable, repeatable practices To ensure project success, the project team should perform the following: o Select appropriate processes required to meet project objectives o Use a defined approach that can be adapted to meet requirements August

7 o Establish and maintain appropriate communication and engagement with stakeholders o Comply with requirements to meet stakeholder needs and expectations o Balance the competing constraints of scope, schedule, budget, quality, resources, and risk to produce the specified product, service or result Activities that should be performed by the project management team (i.e., TOD) include: o Develop, review, analyze, and understand the scope including the project and product requirements, criteria, assumptions, constraints, and other influences related to a project, and how each will be managed or addressed within the project o Measure and monitor the project s progress and take appropriate action to meet project objectives Cause The process to migrate and convert toll data between the legacy vendor and the current vendor did not work effectively to meet contractual and transitional timelines as originally expected by TxDOT. This included a complex conversion of 8 years of tolling data from a multiple account structure to a single account structure to help drive efficient statement reporting for toll customers. Without fully implementing an independent TOD validation protocol for quality control or quality assurance responsibilities, TOD primarily relied on the vendor to self-monitor and self-report on customer service and billing activities from their data. Evidence The audit work performed was to substantiate the timing and transparency of when issues were detected, who was involved in determining corrective actions, root causes, and the escalation path used to notify TxDOT management and administration. Audit work concluded that limited documented information was created or retained to substantiate who was involved in the discussions or who took accountability for actions that were to follow in resolving the operational and reporting issues. A review of TOD s management processes over system implementation and operations identified the following: Limited documentation to support key decisions made during the project, (i.e., approval of extension of time to complete milestones, the determination of the vendor's completion of individual deliverables within a milestone, and approval to process tolling customer billings), could be provided No documented operational design existed for process or procedures to effectively evaluate achievement of system implementation, Customer Service Center (CSC) operations, and timely transaction processing goals o Preliminary key performance indicator (KPI) monitoring techniques used were drafted after contract commencement by the vendor and had not been finalized or enforced, as of audit fieldwork completion August

8 TOD provided no documentation of a process or procedures for a Quality Control/Quality Assurance protocol which would be used to validate tolling data and performance indicators when provided Limited documentation demonstrating TOD s monitoring of the third-party vendor s performance in integration coordination and support of toll operations In reviewing and evaluating system implementation contractual milestones, the following were noted: Required system implementation tests for operations commencement, including the site acceptance test, system integration test, and parallel operations test, were not completed as scheduled resulting in delayed implementation of go-live from the original contract date of March 28, 2014 until July 9, 2014 The Operations Commencement Milestone was delayed for three months leading up to the go-live date; this resulted in additional fees and forgiveness of damages to the previous toll operation s vendor to ensure continued operations. System Final Acceptance scheduled for June 2014 had not occurred, resulting in continued delays in TOD s ability to monitor vendor performance and to independently assess data integrity. Management Action Plan (MAP): Rick Nelson, Director, Toll Operations Division MAP 1.1: Project Management Processes TxDOT will implement Project Management Institute (PMI) PMBOK guideline usage by TxDOT, consultants, and vendor to ensure a well-designed process is implemented and becomes standard operating procedure for contract and project management. October 15, 2015 Roland Silva, Toll Operations Division MAP 1.2: Standard Operating Procedure Development - Phase 1 TOD will continue its development of TOD staff Standard Operating Procedures (SOPs) which will address each TOD Subject Matter Expert s (SME) specific oversight and monitoring activities. SOPs will define monitoring and process review of operational activities and Key Performance Indicators. Given the work effort required to complete this MAP and the impending Operational Observation Period (OOP) schedule, TOD will use the OOP to understand and document its KPI monitoring SOPs. Therefore, KPI related SOPs will be completed during Phase 1 and the remaining operational SOPs will be completed in Phase 2. August

9 Action Completed MAP 1.3: Standard Operating Procedure Development - Phase 2 TOD will continue its development of TOD staff SOPs which will address each TOD SME s specific oversight and monitoring activities of specific operations processes. November 15, 2015 Brian Smallwood, Director, Operational Excellence Toll Operations Division MAP 1.4: Verification & Validation Processes TOD will expand its verification/validation activities to include process audits by TOD operational area SMEs which will be subject to review and audit by the TOD QA/QC Team. Activities identified on MAP 1.2 monitoring SOPs will be audited periodically according to a TOD QC Audit Schedule and subject to regular spot checks. Results and any recommendations will be reported to the Director of Operational Excellence and Toll Operations Director when needed and/or on a quarterly basis. In the interim, TOD will continue its monitoring of identified key processes and coordinate appropriate actions with vendor through existing regularly scheduled Issues Review, Release Planning and QA/QC meetings. October 15, 2015 Rick Nelson, Director, Toll Operations Division MAP 1.5: Independent 3rd party review of operations TOD will work with an independent consultant to provide an independent information technology assessment of contract management and review of operations to provide specific areas of focus to improve both. Consultant will also provide an independent validation of vendor contract and performance by implementing tools using Tableau and Crystal Reports. These tools will be owned and used by TOD to monitor contract performance after the initial assessment. December 15, 2015 August

10 MAP 1.6: Additional PMO process Third party PMP certified consultant implementation of a project matrix detailing approved projects, project owners, resources assigned, percentage of employees time assigned to project, project approved schedule by budget assigned and expended by month. This matrix will include all planned and approved activities for the next 18 months and project and/or contract managers will report monthly at TOD staff meetings on these projects. October 15, 2015 Linda Sexton, Deputy Director, Toll Operations Division MAP 1.7: Atkins PMO Certified Managers TOD will direct their third party consultant in writing to provide the PMP certified project manager(s) required by contract and to adhere to PMBOK guidelines. September 15, 2015 MAP Owners: Rick Nelson, Director, Toll Operations Division Linda Sexton, Deputy Director, Toll Operations Division MAP 1.8: TOD Organization TOD will evaluate its present organizational structure and provided a recommendation to the TXDOT CFO of any needed changes required to address unsatisfactory ratings noted in this Internal Audit Report. November 15, 2015 MAP 1.9: TxDOT PMO Certified Managers TOD will require all identified TOD project managers to obtain PMP certification and use PMBOK guidelines in managing projects. Phase 1 certification of three TxDOT project managers will be completed by December 15, Phase 2 will provide a plan to certify all remaining TxDOT TOD project managers. February 15, 2016 August

11 MAP Owners: Rick Nelson, Director, Toll Operations Division Teresa Lemons, Toll Operations Division MAP 1.10: Procure Project Management Software Develop action plan to procure and implement third party software for Project Management designed around the PMBOK guidelines to provide a dashboard/score card using both qualitative and quantitative information on all approved tasks and projects. Said software shall provide a transparent dashboard/score card and shall be accessible 24 x7 to senior management of TOD and any other users approved by TxDOT CFO. January 15, 2016 Teresa Lemons, Toll Operations Division MAP 1.11: TOD Budget Officer Recruit and hire an individual who is a Certified Public Accountant to fill the Budget Officer position to provide Generally Accepted Accounting Principles compliance guidance to TOD. September 15, 2015 August

12 Finding No. 2: Performance Measure Achievement Condition Achievement of contractual performance measures by the vendor in categories such as Customer Service Center (CSC) Operations, CSC Information Systems, and Timely Processing, were not being consistently monitored or evaluated by Toll Operations Division (TOD). Performance monitoring was scheduled to start 60 days after system implementation. Effect/Potential Impact There was no assurance that the vendor was meeting performance goals in accordance to the contract. Additionally, performance problems or issues that would be identified with the proper assessment of performance measures may not be timely addressed and resolved. Criteria The vendor contract requires payment of damages for failure to meet various performance measures related to the CSC Operations, CSC Information Systems, and Timely Processing. Texas Government Code 2262 requires that state agencies comply with the State of Texas Contract Management Guide (Guide) for monitoring contract performance. The Guide provides guidelines on performance monitoring including the development of performance measure deliverables that clearly outline the methods and procedures to measure results. Cause TOD has elected to not calculate or validate performance measure achievement until all milestones are completed. Currently, there is a reliance on the vendor to self-report performance measurement results with no documented or independent TOD validation protocol. Evidence A review of 10 of 27 contractual performance measures for CSC Operations, CSC Information Systems, and Timely Processing indicated the data used to calculate the measures could not be independently validated as all contractual milestones had not been completed by the vendor. As of the February 28, 2015, reports requested for the 10 performance measures listed in the table below were either incomplete or the data pulled from the system and used in the calculation could not be re-performed to assure reliability. Those performance measures, which are tied to system implementation milestones that are not yet complete, include: August

13 Performance Measure Damages Average Speed to Answer Calls answered by $5,000 per month live agents within 60 seconds, 85% of time and within 120 seconds, 95% of time Wait-time to Answer - Calls coming into the $5,000 per month Interactive Voice Response (IVR) requesting a live agent must have a maximum wait time of 5 minutes, 99% of time Escalations - A maximum of 10% of calls $5,000 per month answered by a live agent are allowed to be escalated. IVR - 25% of calls received, handled through IVR $10,000 per month Inquiries, Requests, and Disputes - Responded $5,000 per month with findings and/or resolution within 3 days, 85% of time, and within 5 days, 100% of time. Statements - Mailed/ ed to customers $5,000 per month within 5 days, 97% of time and within 10 days, 3% of time Website - Excluding scheduled downtime, $1,000 per hour website available 99% of time Timely Notification of TxTag Transactions within Variable pricing of $ $0.050 per paid 60 calendar days toll transactions based on the total number of Timely Notification of Video Transactions within 60 calendar days Timely Notification of Violation Transactions within 60 calendar days Management Action Plan (MAP): transactions posted in the month $0.31 per paid toll transaction based on the total number of transactions posted in the month $0.325 per paid toll transaction based on violation transactions (pre-collection) Brian Smallwood, Director of Operational Excellence Toll Operations Division MAP 2.1: Complete Implementation Milestones TOD will continue to coordinate with the vendor to identify and prioritize the resolution of incomplete implementation milestones. TOD is meeting with the vendor twice weekly to identify issues impeding implementation and to prioritize the evaluating of solution releases through Issues and Release Planning meetings. October 15, 2015 MAP 2.2: Complete Key Performance Indicator (KPI) Reports TOD will continue to evaluate KPI reports available and drive this report review effort, including validation and approval of vendor performance reporting. TOD will deliver a formal August

14 report Validation and Release process document to the vendor to facilitate the validation of the KPI reports. TOD currently meets with the vendor's Technical Reporting team daily to resolve KPI report issues and prioritize their report fixes or enhancement necessary for accurate and complete KPI reporting. A meeting will also be scheduled targeted at gaining a more comprehensive understanding of Vector 4G data and database relationships to facilitate report and data validation. November 15, 2015 Rick Nelson, Director, Toll Operations Division MAP 2.3: Milestone Documentation TOD will request all vendors provide action plans in writing to address missed or late milestone dates. Vendor action plans will require a fully loaded resource schedule so TOD can monitor vendor s effort in meeting milestones. Action Completed MAP Owners: Rick Nelson, Director, Toll Operations Division Linda Sexton, Deputy Director, Toll Operations Division MAP 2.4: System Monitoring Improvements TOD will implement third party software to automate lane audits and monitor system performance. December 15, 2015 MAP Owners: Linda Sexton, Deputy Director, Toll Operations Division Brian Smallwood, Director of Operational Excellence, Toll Operations Division MAP 2.5: Interim Performance Reporting TOD will calculate performance measure achievement by using statistics provided by the vendor and independently evaluate the data to assess damages. This will be done until such time as required reports are approved by TxDOT through Operational Observation Period and available for use. November 15, 2015 August

15 Summary Results Based on Enterprise Risk Management Framework Rating Assessment Grid Exemplary Satisfactory Needs Improvement Unsatisfactory Closing Comments The results of this audit were discussed with management of Toll Operations Division and TxDOT Administration. We appreciate the assistance and cooperation received from the management and employees contacted during this audit. August