Harmonized Risk Scoring-Advance Trade Data Internal Audit Report

Harmonized Risk Scoring-Advance Trade Data Internal Audit Report March 2011

Table of Contents Executive Summary...2 1.0 Introduction...4 1.1 Background...... 4 1.2 Risk Assessment... 5 1.3 Audit Objective and Scope... 6 1.4 Approach and hodology... 6 1.5 Audit Criteria... 6 1.6 Statement of Assurance...... 6 2.0 Audit Opinion...6 3. 0 Findings, Recommendations and Action Plans......7 3.1 Software Development...... 7 3.2 Technology Implementation... 9 4. 0 Other Matters of Interest......10 4.1 Communication...... 10 4.2 Funding and Scheduling Issue... 11 5.0 Management Response...11 Appendix A: Audit Criteria...12 Appendix B: List of Acronyms...14 i

Executive Summary Background The Harmonized Risk Scoring-Advance Trade Data (HRS-ATD) audit is a System Under Development (SUD) audit, approved by the Canada Border Services Agency (CBSA) Audit Committee as part of the Three-year Risk-based Audit Plan for Fiscal Years 2009-10 to 2011-12. The objective of HRS-ATD is to strengthen the CBSA s ability to identify and assess the risk associated with marine shipments of commercial goods destined for Canada. The harmonized aspect of the project refers to harmonizing Canada s risk-assessment and targeting methods with the standards established by the World Customs Organization and the United States Customs and Border Protection. The HRS-ATD project includes seven components. At the time of the audit, two components had been completed: Hybrid 1 (H1), consisting of new risk-scoring algorithms and implemented in September 2009; and U.S. Marine In-Transit (USMIT), consisting of a new process for receiving data on marine containers in-transit from the United States to Canada and implemented in October 2010. HRS-ATD was approved in June 2008 with a scheduled completion date in fiscal year 2010 2011. Its total allocated budget was $31 million. Of note is that the forecast total actual spending at the end of fiscal year 2010 2011 was $22 million, and carryover of the remaining funds to continue project work was not approved. Funding and scheduling options for completing the remaining HRS-ATD components are being addressed by project management (see Section 4.2). Significance of this Audit This audit is of interest because the HRS-ATD project represents a new means of assessing the risk of goods coming into Canada. This activity is central to fulfilling the CBSA s mandate and, accordingly, it is important that senior management have assurance on the extent to which the processes for developing and implementing this project were adequate. Objective and Scope The objective of the audit was to assess the adequacy and appropriateness of the CBSA s processes for managing both the development of HRS-ATD and the integration of the software and hardware products flowing from HRS-ATD with the CBSA s existing computer technology. The H1 component was implemented under a different project organization structure than USMIT, and followed management processes that have since been revised for the remaining components. Consequently, in meeting this audit objective, the audit focused on HRS-ATD s USMIT component as it was more representative of the current development processes used by 2

HRS-ATD. As no changes were needed in the computer hardware associated with USMIT, the audit included examination of the processes to implement USMIT on the existing hardware. Audit Opinion The audit found that the processes which the HRS-ATD project followed for managing the development and implementation of the USMIT component were adequate and appropriate. Key Findings HRS-ATD s processes for developing USMIT were consistent with the CBSA s Major Project Governance Framework, and USMIT was successfully developed and implemented. The USMIT component used standard technology and did not affect the CBSA s other computer technology or processes. Of the nine criteria considered relevant to this audit, the project met seven and partially met the other two. Our observations were mainly positive, so we make no recommendations in this report. However, the audit team did note opportunities for improvement. These matters posed little risk to USMIT. However, they warrant management s attention as they could affect the scope and delivery schedules for the future components associated with HRS-ATD. Accordingly, they are covered under Other Matters of Interest. Statement of Assurance This audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada. 3

1.0 Introduction 1.1 Background The Canada Border Services Agency (CBSA) Audit Committee approved an audit of the Harmonized Risk Scoring-Advance Trade Data (HRS-ATD) project as part of the Three-year Risk-based Audit Plan for Fiscal Years 2009-10 to 2011-12. The CBSA is looking to effectively push the border out to ensure that serious threats to Canada s health, safety and security are identified and intercepted before they approach or cross Canada s physical border. To this end, the CBSA has carried out various complementary activities, including the HRS-ATD project. Accordingly, the objective of HRS-ATD is to strengthen the CBSA s ability to identify and assess the risk associated with commercial marine shipments bound for Canada, and to target high-risk shipments to prevent them from entering this country. The harmonized aspect of the project refers to harmonizing, to the extent possible, Canada s risk-assessment and targeting methods with the standards established by the World Customs Organization (WCO) and the United States Customs and Border Protection (U.S. CBP). HRS-ATD has been built on the successful marine component of the Advance Commercial Information program. This program requires carriers to electronically transmit data on marine cargo to the CBSA 24 hours before that cargo is loaded onto a ship in a foreign port. HRS-ATD will improve the Agency s ability to identify and target risky commercial marine shipments by: harmonizing, as noted above, risk-assessment and targeting methods with the standards established by the WCO and the U.S. CBP; incorporating an expanded set of risk indicators; and incorporating new sources of advance trade data. The HRS-ATD project consists of seven components. At the time of the audit, two components had been completed: Hybrid 1(H1), consisting of new risk September 2009; and scoring algorithms and implemented in U.S. Marine In-Transit (USMIT), consisting of a new process for receiving data on marine containers in-transit from the United States to Canada and implemented in October 2010. The five remaining components to be developed are: Advanced Trade Data (ATD), for capturing Advance Trade Data Set (ATDS). Container Status Messages (CSM), which will increase the data collected in the risk assessment process and will allow targeters to make more informed pre-load and pre-arrival marine decisions. Determining and documenting historical trade patterns using existing and new commercial information to develop new risk indicators. The trade pattern information 4

will include data related to routing, trade chain partners, the commodity, importer and combinations of these elements. Establishing contracts with corporate and trade data providers. The corporate and trade profile data will be used where the CBSA has determined that certain trade chain partner involved, such as shippers and consignees, in the importing of the cargo in the marine mode are not known to the CBSA. Implementing new scoring algorithms into TITAN and ACROSS to enhance the CBSA s marine container tracking and targeting capabilities, which are comparable to that of the United States. While not completed, some progress had been made in developing these components. For example, hardware and software to deliver three of the remaining components dealing with continuing analysis of risk indicators had been purchased, and some supporting processes had been developed. HRS-ATD was approved in June 2008 with a scheduled completion date in fiscal year 2010 2011. Its total allocated budget was $31 million. Of note is that the forecast total actual spending at the end of fiscal year 2010 2011 was $22 million, and carryover of the remaining funds to continue project work was not approved. Funding and scheduling options for completing the remaining HRS-ATD components are being addressed by project management (see Section 4.2). USMIT is not a separate, independent application; rather it is a feature being added to four large existing applications. 1 1.2 Risk Assessment The risk assessment carried out when planning this audit identified the following key risks: Development Process Changes in roles and responsibilities for developing the software for HRS-ATD were viewed as a risk that could affect the development of future HRS-ATD components. Technology Integration The Agency has experienced both delays in obtaining data-mining hardware, and difficulty in reaching the service levels necessary to improve its risk assessment for marine shipments. This situation has given rise to the risk that computer hardware will not be available in time to allow the Agency to get the necessary software up and running in a timely manner. The examination phase of this audit determined that the risk associated with the Development Process (refer to Section 3.1, Criterion 1.3) was not a factor for USMIT. However, the delivery of future components of HRS-ATD could be affected if the role of the sponsoring organization with respect to requirements is not clarified (refer to Section 4.1). For the risk related to Technology Integration, the audit determined that this risk did not impact USMIT as it used 1 These applications are: the Customs Electronic Commerce Platform (CECP), TITAN (the new name for the Advance Commercial Information Risk Assessment application), the Accelerated Commercial Release Operations Support System (ACROSS), and Commercial Risk Scoring and Assessment (CRSA). 5

existing hardware not new hardware. Technology Integration risks associated with the five HRS-ATD components still to be completed will be addressed by project management in the revised funding and scheduling plans referred to in Section 4.2. 1.3 Audit Objective and Scope The objective of the audit was to assess the adequacy and appropriateness of the CBSA s processes for managing both the development of HRS-ATD and the integration of the software and hardware products flowing from HRS-ATD with the CBSA s existing computer technology. The H1 component was implemented under a different project organization structure than USMIT, and followed management processes that have since been revised for the remaining components. Consequently, in meeting this audit objective, the audit focused on HRS-ATD s USMIT component as it was more representative of the current development processes used by HRS-ATD. As no changes were needed in the computer hardware associated with USMIT, the audit included examination of the processes to implement USMIT on the existing hardware. 1.4 Approach and hodology The audit gathered evidence by conducting interviews, reviewing documentation and assessing the HRS-ATD system-development and software-integration process. The audit: interviewed selected project personnel to assess the development processes project, specifically to deliver USMIT; on the HRS-ATD interviewed technical management personnel to assess the CBSA s readiness to deal with the new technology, the overall technology configuration management, and plans to manage USMIT s impact on computer software and hardware related to this component; and reviewed project documents to assess the degree to which HRS-ATD project management practices aligned with industry practices and applicable Treasury Board (TB) and Agency policies. 1.5 Audit Criteria The detailed audit criteria are presented in Appendix A of this report. 1.6 Statement of Assurance This audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada. 2.0 Audit Opinion The audit found that the processes which the HRS-ATD project followed for developing and implementing the USMIT component were adequate and appropriate. 6

3.0 Findings, Recommendations and Action Plans 3.1 Software Development This section looks at the extent to which the software for HRS-ATD was developed in accordance with the Agency s standards for this area. Using a formal system-development process with specific milestones is accepted as a best practice and is essential for managing risk in developing software applications. Adhering to a standard means that all projects will follow a similar, proven process; that management will know what to expect; and that terminology will not be confusing to participants. This section presents the findings relating to audit criteria 1.1 to 1.6, listed in Appendix A. The System Development Life Cycle Standard (Criterion 1.1) This Standard is an industry term for a well-controlled software-development process. The CBSA Major Project Governance Framework 2 is the Agency s standard approach to managing major projects. It states that projects are to follow a Project Management Life Cycle approach that includes six phases, with formal approvals after each. The USMIT component of the HRS-ATD project (a major project) met this criterion, and it followed the Agency s framework in most respects. We noted that in the case of its USMIT component, the project had obtained only informal sign-off approvals at the end of each phase, rather than the formal approvals that the framework calls for. The framework is designed to ensure among other things that systems will meet all business requirements. Therefore, any deviation from the standard framework creates a potential risk that the project may not meet future business requirements. The audit noted that the CBSA had already begun to review and re-develop the governance structure for major projects with a view to strengthening the formal approval process. When the new structure has been approved, HRS-ATD will be required to follow it. Requirements Definition (Criterion 1.2) This criterion calls for a formal process for ensuring that business, technical, and security and privacy requirements have been agreed upon and clearly documented. As well, the documented requirements should be traceable to the final software product to ensure that it reflects these requirements. This process helps to ensure that a system will contribute to fulfilling an organization s business requirements. The audit found that the project met these criteria. Requirements had been clearly specified, documented and met. In examining and tracing a sample of business requirements, we found that 2 CBSA Major Project Governance Framework Version: v1.3, November 6, 2007, document maintained by the Enterprise Project Management Office, Information, Science and Technology Branch 7

they were reflected in the design of the software. Staff from the project team had validated the way in which USMIT software had met documented business requirements. All problems had been resolved before implementation. Communication among stakeholders (Criterion 1.3) A key means of controlling risk when following a standard system-development approach is to determine where and when essential communication must take place among all stakeholders. According to this criterion, systems developers and clients should communicate clearly with each other on the design of a system, and project management, users and the project sponsor should sign off on the design. The project generally met this criterion. The audit found that communication with respect to defining requirements for the USMIT component was adequate. Subsequently, the organizational structure under which the Agency had defined the requirements for USMIT changed. While these changes resulted in a lack of clarity on the respective role of the sponsor and project management for USMIT development and implementation, this lack of clarity did not affect USMIT. In order to ensure that future HRS-ATD components meet all business requirements, it would be beneficial to clarify the role of the sponsoring organization. Please refer to Section 4.1 for further discussion. Software Construction (Criterion 1.4) Once requirements have been determined, software should be developed that reflects the agreedrequirements, and meet relevant standards. upon and documented business The project met this criterion. The development of business requirements and software conformed to the Major Project Governance Framework. Software Assurance (Criterion 1.5) The criterion for this area requires a Quality Assurance (QA) mechanism for software which includes comprehensive testing and ensures that this software will ultimately meet business and security requirements. The assurance process includes testing to verify, most importantly, that the software does what it is intended to do (functionality) and also that the software meets performance and other requirements. The project partially met this criterion. The audit was advised that work was in process to strengthen software assurance controls. Testing was well managed and was successfully completed. The scope of testing included software functionality and other areas such as the time it takes for the software to respond to enquiries, the time taken to recover in case of failure, and performance. We were advised that the Agency s Information Technology (IT) quality management organization is implementing a more rigorous process to ensure the readiness of all systems, not 8

only from a testing perceptive, but, in terms of ensuring appropriate documentation is completed and approved. A standard assurance process includes ensuring that proper security requirements have been documented, and certifying that they have been met. The audit found that a Threat and Risk Assessment (TRA) for USMIT had not been prepared. Information from a TRA is an input to certifying that a system has met all security requirements. As USMIT uses existing hardware and software, its TRA analysis was dependent on a review and update of the TRA analyses for the existing systems. This work had yet to be done. In the meantime, an interim authority to operate USMIT had been approved. This authority was scheduled to expire in January 2011. We note that the risk had been assessed as Medium until this work could be completed. We were advised that the Agency s IT security organization expects to complete TRA updates by April 2011, and that the interim authority has been extended until September 30, 2011, by which time the work on the TRA will be completed. Software Implementation (Criterion 1.6) According to this criterion, an organization should have a process for implementing software applications which ensures that new software is integrated into existing operations in a controlled manner. The CBSA met this criterion, and the audit found that the implementation of USMIT was generally adequate. 3.2 Technology Implementation Projects must consider how their software could affect other existing computer hardware and software. This section focuses on the degree to which project deliverables conformed to the CBSA s technical standards and processes. The sub-sections below align directly with audit criteria 2.1 to 2.3 listed in Appendix A to this report. Central Infrastructure Processes (Criterion 2.1) This criterion would require HRS-ATD to adhere to the Agency s processes and standards to ensure that any new hardware and software is implemented in accordance with established acquisition and maintenance processes. The project met this criterion. We found that the Agency had a process for managing the implementation of new hardware although, in the case of USMIT, no new hardware was involved. The criteria for this audit included two others (Criteria 2.4 and 2.5), which related to planning for, acquiring, and implementing and maintaining the technological infrastructure. Since, as noted above, USMIT did not require any new hardware, these two criteria were not applicable. 9

IT Capacity and Disaster Prevention (Criterion 2.2) IT deliverables have the potential to affect either the capacity or security of the Agency s existing computer systems. Therefore, according to this criterion, the HRS-ATD project should adhere to the Agency s IT security policies and standards. The project partially met Criterion 2.2. With respect to capacity, in implementing USMIT, HRS-ATD followed the Agency s standard processes. Since USMIT is not an independent application, it was not necessary to change the technology architecture. However it was necessary to analyse whether the existing computer operations had enough capacity to handle the increased workload that would result from integrating USMIT with existing systems. The CBSA did this analysis, which indicated that there was no need for any specific acquisitions to support USMIT because of the additional volume of data. Regarding security, a key element is disaster prevention. The requirements for USMIT included one relating to restoring the application should a disaster occur. However, there was no disasterrecovery plan in place for at least one application (ACROSS) relating to USMIT. The recent CBSA audit of Business Continuity Planning resulted in a management plan to address weaknesses in this area. The Agency s Data Centre Recovery Project, started in fiscal year 2010 2011 and expected to be completed in fiscal year 2013 2014, will look at the CBSA s ability to recover from a disaster and maintain business continuity. This project is expected to be completed in four years time. In the meantime, however, there is a continuing residual risk that a disaster or other significant event could interrupt essential CBSA services. Management has accepted this risk. Development and Test Environments (Criterion 2.3) To minimize the potential risk to ongoing CBSA computer programs and operations, the HRS-ATD project should be able to carry out efficient, effective tests of its computer software and hardware. This criterion was met. The CBSA has various test environments for thoroughly testing new applications and infrastructure. The approach to testing was designed to minimize any potential impact during implementation. 4.0 Other Matters of Interest While carrying out this audit, the audit team noted certain issues that did not directly affect the delivery of the USMIT component of HRS-ATD. However, in our opinion, these issues were potentially important enough to warrant management s consideration. 4.1 Communication As noted under the heading Communication among stakeholders, a new organizational structure had replaced the one in effect when USMIT was under development. Under the original structure, business requirements had been developed, documented and approved by a committee of sponsors from all stakeholder branches within the CBSA. While a new sponsor had been 10

identified under the new structure, the respective roles of the sponsor and project management were unclear in this area. Consequently, some confusion existed over responsibility for defining business requirements. The audit team found that the HRS-ATD project team and the sponsor representative each believed that it played a key role in defining and approving business requirements. In interviews, staff from the new project sponsor indicated they were unclear of their role in ensuring that the project would ultimately meet the requirements. This lack of clarity did not affect USMIT because its requirements had been developed and approved under the previous organizational structure. However, if this lack of clarity persists, there is a risk that future components may not meet all business requirements. 4.2 Funding and Scheduling Issue The audit was advised by HRS-ATD project management that approximately $22 million of the $31 million HRS-ATD budget would be expended by March 31, 2011, and that carryover of the remaining funds to continue project work was not approved. Management has advised that a funding strategy and revised schedule for implementing the advanced risk assessment functions, which account for three of the five remaining components, has now been developed, and that work was still progressing on updating scope, funding plans and delivery schedules for the two remaining components (ATD and CSM). 5.0 Management Response Management acknowledges and thanks the audit team for noting the matters of interest. With respect to the identified matters of interest noted in Section 4: - Section 4.1 Communication: Under the new Agency organization and using the new CBSA Project Governance Framework for Major Projects, a new governance structure framework for projects has been developed and implemented. This governance framework, including sponsorship details, was approved in November 2010 and is followed by HRS-ATD. The Risk Assessment Directorate has been identified as the sponsor for HRS-ATD; and - Section 4.2 Funding and Scheduling Issue: Action is being taken to develop options for the CSM component. Analysis is expected to be completed by March 2011 and will be submitted for formal approval via project governance committees. The ATD component will be delivered as part of the emanifest project with implementation targeted for fiscal year 2013 2014. The plan for ATD is being prepared as part of the planning of the emanifest project. The new plan for the emanifest Project will be completed by November 2011. Also, as noted above, with respect to the security requirements for USMIT (see Section 3.1, Criterion 1.5), the IT security organization expects that TRA updates will be completed by April 2011 and that the interim authority has been extended until September 30, 2011, by which time the work on the TRA will be completed. 11

Appendix A: Audit Criteria The audit criteria used for the HRS-ATD audit were: Line of Enquiry Audit Criteria 1. Development Process 1.1. System Development Life Cycle Standard. The HRS-ATD Project has a software development and acquisition standard that is adequate for the complexity of the project. 1.2. Requirements Definition. A formal process exists t o ensure business, technical, and security/privacy requirements to achieve the expected outcomes of the HRS-ATD Project are identified, prioritized, specified and agreed upon. 1.3. Communication during the Development and Delivery Process. Business solution designs have been effectively communicated and articulated between systems development and clients, and been signed off by project management, users and project sponsor representatives. 1.4. Software Construction. Automated functionality is being developed in accordance with design specifications, development and documentation standards, QA requirements, and approval standards. Software components are seen as configurable items and base-lined. 1.5. Software Assurance. Assurance tasks needed to support the accreditation of new or modified systems that meet externally defined requirements for accreditation and/or certification have been identified, including a test environment and user involvement. 1.6. Software Implementation. The project has an implementation and fallback/backout plan. Processes exist and the authority has been established to approve releases on behalf of or representative of project sponsors. Partially Not Partially met 2. Infrastructure and Technology Transition 2.1. Central Infrastructure Processes. The HRS-ATD Project adheres to CBSA central processes and standards to ensure that installation and maintenance of system software is in accordance with the acquisition and maintenance framework for the technology infrastructure. 2.2. IT Capacity and Disaster Prevention. The HRS-ATD Project adheres to Agency s IT security policies and standards. 2.3. Development and Test Environments. There are development Partially met 12

Line of Enquiry Audit Criteria and test environments established to support effective and efficient testing of infrastructure components. 2.4. Acquisition, Implementation and Maintenance of Technological Infrastructure. The HRS-ATD Project has produced a strategy and plan for the acquisition, implementation and maintenance of the technological infrastructure that meets established business functional and technical requirements. 2.5. Technology Transition Plan. A Transition Plan exists that identifies and documents all technical, operational and usage aspects for implementation of technical components and the transfer of information to stakeholders. Partially Not N/A N/A 13

Appendix B: List of Acronyms Acronym ACROSS ATD CBSA CECP CRSA CSM EPA H1 HRS-ATD IT SUD TB TITAN U.S. CBP USMIT WCO Description Accelerated Commercial Release Operations Support System Advanced Trade Data Canada Border Services Agency Customs Electronic Commerce Platform Commercial Risk Scoring and Assessment Container Status Messages Effective Project Approval Hybrid 1 Release Harmonized Risk Scoring-Advance Trade Data Information Technology System Under Development Treasury Board New name for the Advance Commercial Information Risk Assessment application U.S. Customs and Border Protection U.S. Marine In-Transit World Customs Organization 14