A Risk-based Approach to Performance Auditing

CCAF-FCVI Fellow 2009/2010 A Risk-based Approach to Performance Auditing Strategic Paper by Levina Rusk Kishimba TANZANIA CANADA

Table of Contents Page Acknowledgements Executive Summary v vii Introduction 1 Background 1 Objective 1 A Risk-based Approach to Performance Auditing 2 Context 2 Problems to be addressed 2 Rationale for using one-pass planning in the office 2 Challenges ahead 3 Description of the Strategy 4 Phase 1 Create awareness and share knowledge 4 Phase 2 Train performance auditors 4 Phase 3 Develop OPP approach 4 Phase 4 Learn by doing 5 Proposed Approach 5 Phase 1 Briefing sessions with management and staff 5 Phase 2 In-house training 5 Phase 3 Workshops 6 Phase 4 Pilot project for the selected entity 6 Evaluation of the Strategy and Next Steps 6 Conclusion 7 Appendix A Project Implementation Schedule 9 Appendix B Steps to Preparing a One-Pass Plan 11 Bibliography 14 iii

Acknowledgements Praise is to the Lord God who always guides my every step and keeps me in the centre of his will. This strategy paper is the result of a nine-month International Fellowship Program I attended in Canada from August 2009 to May 2010. The fellowship was awarded by the Canadian Comprehensive Auditing Foundation (CCAF) with funding provided by the Canadian International Development Agency (CIDA). Special thanks and appreciation to CCAF; CIDA; the Auditor General of Canada, Ms. Sheila Fraser; and the staff of the Office of the Auditor General of Canada (OAG Canada) for organizing and running the program. I would also like to express my gratitude to OAG Canada s Canada Revenue Agency Performance Audit Team (Group 7, Team 8), the people who ran the Office s training program, and International Relations staff for the support they gave me in developing my strategy paper and gaining a lot of knowledge in performance auditing. My participation in this fellowship program would not have been possible without the vision of the Controller and Auditor General of the United Republic of Tanzania, Mr. Ludovick Utouh, in improving the knowledge and skills of his staff. I sincerely thank my mentor, Mr. Ronald Bergin, Principal of Strategic Planning at OAG Canada, for sharing his time and insight throughout the development of my strategy paper. I am very grateful to my darling husband, Timothy, who was always there to support and encourage me during my stay in Canada. Many thanks also to all 2009 2010 CCAF International Fellows for their friendship. It was nice spending time with them. Finally, I wish to extend my deep and lasting appreciation to all who, in one way or another, helped me throughout the fellowship program. Levina Rusk Kishimba May 2010 v

Executive Summary This strategy paper proposes how the National Audit Office of Tanzania (NAOT) will adapt and implement the Office of the Auditor General of Canada s one-pass planning approach to selecting performance audits. One-pass planning (OPP) is a risk-based audit planning approach which focuses on how well an entity is managing its major risks rather than on areas of suspected weakness. Preparing one-pass plans for performance audits will provide assurance to Parliament and other stakeholders that NAOT is following a systematic and independent, risk-based and objective approach to selecting the areas for audit. This will demonstrate that it is efficiently using the available resources. In addition, this approach will ensure that any entity deemed to be significant has a current, multi-year plan based on a high level of risk analysis. This approach will be implemented incrementally. It will begin with a briefing for management and staff, followed by training for an initial core of staff on preparing and applying an OPP approach to audit selection. Current audit structures will be adopted to suit the OPP process, and then a pilot survey will be carried out. The new approach will be monitored, evaluated, and adjusted throughout the implementation process. vii

Introduction Background 1. In the past, long-term entity planning in the Office of the Auditor General of Canada (OAG Canada) focussed on determining the value-for-money audit priorities. Planning was carried out from the perspective of making a difference for Canadians by identifying areas that likely needed improvement. As a result of a project entitled Advancing Audit Practices, approved in June 2001, OAG Canada implemented a more systematic, integrated, and risk-based approach to long-range entity planning, referred to as one-pass planning (OPP). 2. The term one-pass signifies that the knowledge of both performance and financial auditors is brought together in one analysis at the beginning of the process to help document the auditors understanding of the risks that a particular entity is facing and how well it manages those risks. Near the end of the process, the auditors again consider a combined analysis to determine to what extent they will use financial audits and performance audits to address the issues they wish to audit. 3. The National Audit Office of Tanzania (NAOT) has not yet established a strategic planning process to ensure that only relevant matters involving significant risks are selected for performance audits. Resources allocated for performance audits are always limited, which makes selecting entities or areas to be audited a key issue. Putting into place a process which allows the preparation of one-pass plans for the audit entities would enable the Office to focus its limited resources on areas of greatest risk. Objective 4. This strategy paper proposes the integration of a risk-based approach to selecting potential performance audits in the office of the Controller and Auditor General (CAG) of the United Republic of Tanzania, namely, one-pass planning (OPP). OPP will assist with the selection of audits based on significance and business risks (i.e. risks to the achievement of the entity's objectives). It will use resources more effectively and reduce the time and cost of conducting an audit. This will therefore help NAOT improve efficiency in the planning phase of its performance audits. 5. The main objective of this paper is to provide a strategy which will enable NAOT to implement an OPP approach as a tool for selecting performance audit topics and to integrate the preparation of one-pass plans into the strategic planning process of the office by July 2012. 1

A Risk-based Approach to Performance Auditing Context Problems to be addressed 6. The National Audit Office of Tanzania (NAOT) does not have a formal way to select its performance audits. The audits are selected based on complaints from the public and problems identified by the media. There are various risks associated with this approach, including: choosing topics that are sensational but not fundamental to a government department s mandate; choosing topics that are sometimes hard to audit, for example, due to a lack of background, difficulty determining scope, or risk of missing important areas; and risking the Office s credibility by picking irrelevant topics for political reasons (not being independent of the political process of the day). 7. Other problems include limited resources allocated for performance audits and a lack of cooperation from auditees. Rationale for using one-pass planning in the office 8. A risk-based approach offers what may be the best way for NAOT to demonstrate independence and objectivity in addressing the problem of deciding what areas, entities, or themes should be chosen for performance audit. 9. As used by the Office of the Auditor General of Canada (OAG Canada), the onepass planning (OPP) approach emphasizes that planning at the entity level should address all mandate areas of OAG Canada simultaneously. It applies to audit planning for programs and functional areas within and across federal departments and agencies. By focusing on areas of greatest risk to entities, OAG Canada is moving toward an assurance-based perspective. This reinforces its goal of addressing the areas of greatest significance to entities and the expectation that its audits can result in a positive opinion about the management systems and practices it examines. Applied to NAOT, this approach could also improve its relationships with auditees. 10. Further, since adopting OPP, OAG Canada is in a better position to assure Parliament that audit efforts have been devoted to areas of highest importance, and that OAG Canada is achieving an appropriate balance of effort with the limited resources available for the audit. It is in a better position to respond when questions are asked about why it did or did not audit certain areas. Similarly, adopting an OPP approach for the audit entities would enable NAOT to focus its limited resources only on areas of greatest risks. 2

Challenges ahead 11. Performance Audit is a small division of NAOT, with only 11 auditors. Success in implementing the approach will involve some challenges that can be overcome in different ways by the implementation committee. The committee will include performance auditors, the selected financial auditors, Development Plans Unit officials, the Controller and Auditor General and the Assistant Auditor General, Value for Money. It s worth noting that the success of the whole process is highly dependent on the support and involvement of NAOT management. The following sections describe expected challenges. 12. Performance auditors are not familiar with risk-based approaches to performance auditing. Identifying and assessing risks are activities central to OPP. None of the performance auditors at NAOT has technical expertise, experience, or training on using risk-based approaches to selecting matters for performance auditing. Making such a fundamental change will be a challenge, and therefore some performance auditors may be reluctant to adopt this new approach. This challenge can be addressed by explaining the benefits of using OPP and training the auditors on how to prepare and use the approach. 13. Ministries, departments, and agencies (MDAs) as well as local government authorities do not identify and assess risks that would prevent them from delivering their programs. OAG Canada uses the auditees own identification of risks as a check in its OPP approach. In Tanzania, MDAs do not identify and assess their risks. To address this challenge, the NAOT performance auditors, with the assistance of MDA officials, will identify and assess the entities risks. 14. Relations between auditees and NAOT performance auditors are strained and currently not conducive to good collaboration. Sharing information and a collaborative spirit are very important to the success of a one-pass plan. Lack of cooperation has been identified as an issue and is being addressed by another Fellow from NAOT. Both projects are running at the same time, which could have represented either another challenge or an advantage. In this case, the Fellow who will be implementing his strategy paper Developing an Auditees Guide to the Performance Audit Process, will be helpful because his paper will help inform the selected pilot project entity about what NAOT will be doing before the pilot project begins. 15. The NAOT does not have expertise yet to train its performance auditors to conduct OPP. The key players at OAG Canada who use OPP are the principals and directors of audit teams, the Strategic Planning and Professional Practices group, and the consultants. To address NAOT s lack of expertise, I will transfer the knowledge I have gained from OAG Canada effectively to all performance auditors. Further, an OPP Team will conduct a pilot study as a training project. In addition, consultations and advice will be applied whenever required. 3

16. OPP takes time and resources. OAG Canada uses between 500 and 600 hours to prepare a one-pass plan for a single department. In addition, it has a total of 103 performance auditors. The NAOT performance auditors represent a tiny portion of the office operations (11 performance auditors out of around 600 employees), which could prompt some there to wonder why the auditors should get the money required to put the OPP approach into place. Also, OPP does not produce visible results in the short term, a condition which usually makes it harder to get funds and resources in light of competing priorities. 17. Nevertheless, NAOT has accepted the implementation of the risk-based approach (OPP) in selecting its performance audit topics, and so funding will not be a fundamental issue. Since the office has a limited number of performance auditors, the approach will be adapted to suit our local conditions. Description of the Strategy 18. The one-pass planning (OPP) approach will be implemented in four phases, each including a number of supporting activities. Phase 1 Create awareness and share knowledge 1. Submit the strategy paper to the Controller and Auditor General (CAG) and the Assistant Auditor General, Value for Money (AAG-VFM) for review and approval. 2. Prepare agenda for the meetings to be conducted. 3. Select financial auditors for the implementation committee. 4. Hold briefings with management, all performance auditors, selected financial auditors, and Development Plans Unit officials, about the approach and the cost involved (human resources, financial, and time). Phase 2 Train performance auditors 1. Prepare and review presentations and other material. 2. Deliver training to performance auditors on preparing an entity s one-pass plan. Phase 3 Develop OPP approach 1. Conduct workshops with the CAG, the AAG-VFM, and all performance auditors to develop the OPP approach for the Office. 2. Form the team of at least three performance auditors (OPP Team) to prepare a one-pass plan for a small entity to be determined. 3. Develop the approach (the one-pass plan format to be used in the Office). 4. Submit the prepared approach to the CAG for approval. 4

Phase 4 Learn by doing 1. Plan a pilot one-pass plan for the selected entity. 2. Prepare and conduct the one-pass plan for the entity. 3. Write the one-pass plan report. 4. Present the one-pass plan report to NAOT management. 5. Finalize the one-pass plan report and submit it to the CAG for approval. 6. Introduce results of the one-pass plan to NAOT management and the implementation committee. 7. Present the approved one-pass plan report to the management of the entity. 8. Prepare for the audit (select the risks to base it on). 9. Undertake a full performance audit for the selected entity based on the selected risks. 10. Integrate the OPP approach into the Office s strategic planning process. Proposed Approach 19. Since its inception in 2005, the NAOT Value for Money Audit Division has been working hard to build capacity and achieve intended objectives through workable budgets and action plans. To ensure that this strategy is smoothly implemented, the OPP Team will frequently consult with NAOT management and hold a number of meetings with them during the project planning stage. Phase 1 Briefing sessions with management and staff 20. I will hold a briefing with the CAG and AAG-VFM to discuss the strategy and its objectives. Then, they will review the implementation schedule and provide their feedback and approval. The AAG-VFM will select financial auditors to be involved during the briefing session and on the implementation committee. I will then review OPP discussion papers and other related material obtained from OAG Canada before preparing an agenda for the coming briefing session. The session will include management and other staff and focus on OPP, its related concepts, and the rationale for adopting and implementing OPP as a risk-based approach to selecting our audits. Phase 2 In-house training 21. I will refer to the knowledge I ve gained and reading material I ve obtained from OAG Canada when preparing presentations on how to conduct OPP. I will review the presentations before starting the in-house training with the auditors. 5

Phase 3 Workshops 22. All performance auditors will be involved in developing the OPP approach to be used in the office. The CAG and the AAG-VFM will also participate in developing the approach. Then, we will hold workshops to develop a simplified approach which will suit our local conditions. We will review and discuss the Integrated Risk Management process and OAG Canada s OPP process to determine what will apply to developing a one-pass plan format for the office. Phase 4 Pilot project for the selected entity 23. The pilot one-pass plan will be used as a training project for learning how to conduct them. Further, it will be helpful in determining whether the developed approach (the adapted OPP process) works, and hence in identifying what adjustments need to be made before integrating it into the Office s strategic planning process and putting it into practice across NAOT. 24. The performance auditors selected to develop the Office s OPP approach (OPP Team) will conduct a pilot one-pass plan using the agreed upon format. Once the pilot project is complete, the team will hold a lessons learned session with management and other staff to identify what worked well and what could be improved. The team will then write an OPP report and forward it to both NAOT management and the surveyed entity. Based on the results of the report, NAOT management and performance auditors can decide on a list of possible audits to be conducted for that particular entity over the next three years, as well as the timing for the audits. 25. Through this exercise, the implementation committee will become familiar with the approach. The approach will then be introduced to other entities (at all levels of the government) to reinforce the understanding of the government-wide audit issues and the role of all players. Evaluation of the Strategy and Next Steps 26. The successful implementation of this strategy will be measured by the completion of a one-pass plan for the selected entity as a pilot project by July 2011; the integration of OPP into the strategic planning process of the office by July 2012; and the number of performance audits that will successfully be completed on time and within budget as well as the quality of performance audit products of the office, effective July 2012. 27. This success will also depend on the support of other stakeholders inside and outside NAOT. Therefore, the Office must ensure that all of its stakeholders are aware of this new approach to its performance audit work. 6

28. Further, efforts will be made to ensure that all activities in each phase of the strategy are completed within two years (from July 2010 to July 2012). 29. The newly developed approach will then be monitored, evaluated, and adjusted on a periodic basis to ensure continuous learning and improve performance audit reporting. Conclusion 30. The National Audit Office of the United Republic of Tanzania faces a complex challenge in selecting its performance audit topics. This makes the Office inefficient during the planning phase of an audit. The huge number of audit entities and the limited resources available for performance audit work makes the task of choosing priority areas to audit very critical. The introduction of one-pass planning as a risk-based approach to selecting matters for audit will enable the office to make optimal use of its resources and provide assurance to the parliamentarians, central and local governments, and the public that the office is fulfilling its responsibilities. 7

Appendix A Project Implementation Schedule Activity Time frame Key player(s) Phase 1 Create awareness and share knowledge 1. Submit the strategy paper to the CAG and AAG-VFM for review and approval June 2010 (3rd and 4th week) Levina R. Kishimba Vacation July 2010 Levina R. Kishimba 2. Prepare agenda for meetings 3. Select financial auditors for the implementation committee 4. Hold briefings with management, all performance auditors, selected financial auditors, and Development Plans Unit officials August 2010 September 2010 (1st week) September 2010 (2nd, 3rd and 4th week) Levina R. Kishimba The AAG-VFM Levina R. Kishimba Phase 2 Train performance auditors 1. Prepare and review presentations and other material 2. Deliver training to performance auditors on preparing an entity s onepass plan October 2010 October 2010 Levina R. Kishimba Levina R. Kishimba Phase 3 Develop OPP approach 1. Conduct workshops to develop the OPP approach for the office 2. Identify the entity and OPP team November 2010 (1st to 3rd week) November 2010 (4th week) The CAG, the AAG-VFM, and all performance auditors The CAG and the AAG-VFM 3. Develop the one-pass plan December 2010 (1st and 2nd week) All performance auditors, the CAG and the AAG-VFM 4. Submit the prepared approach to the CAG for approval December 2010 (3rd week) Levina R. Kishimba 9

Activity Time frame Key player(s) Phase 4 Learn by doing 1. Plan a pilot one-pass plan for the selected entity 2. Prepare and conduct the one-pass plan for the entity 3. Write the one-pass plan report 4. Present the OPP report to NAOT management 5. Finalize the one-pass plan report and submit it to the CAG for approval 6. Introduce results of onepass plan to NAOT management and implementation committee 7. Present the approved onepass plan report to the management of the entity 8. Prepare for the audit (select the risks to base it on) 9. Undertake a full performance audit for the selected entity based on the selected risks 10. Integrate the OPP approach into strategic planning process of the office January to April 2011 January to April 2011 May 2011 June 2011 (1st week) June 2011 (1st week) June 2011 (2nd week) June 2011 (3rd week) June 2011 (4th week) July 2011 to June 2012 July 2012 OPP Team OPP Team OPP Team Levina R. Kishimba Levina R. Kishimba Levina R. Kishimba Levina R. Kishimba OPP Team and other performance auditors OPP Team and other performance auditors OPP Team 10

Appendix B Steps to Preparing a One-Pass Plan The OAG Canada audit teams consistently follow a series of steps in analysing business risks that are critical to the entity s success and reporting the results of the analysis to both the executive committee and the entity. 1. Conduct interviews and review documents This step involves interviewing key officials of the entity and the officials who are charged with its stewardship responsibilities to identify its objectives. Further, the audit team interviews OAG Canada personnel with experience auditing the entity. The audit team then reviews all key documents needed to assist in analysing the risks facing the entity. 2. Document knowledge of the entity The second step in the OPP process involves documenting knowledge of the entity. The audit team records a description of the entity s objectives, responsibilities, and expected results. The team also prepares a summary of the following: interviews completed, the entity s enabling legislation, its other key mandates and financial authorities, its mission and objectives, its strategies to achieve objectives, and its plans for business and operational structure. 3. Prepare the entity risk profile In conducting the risk analysis, the audit team uses a generic government risk model to help identify the most significant internal and external risk factors believed to exist in government. In using the model, attempts are made to capture the most common factors liable to result in hazards or risks that will prevent the entity from achieving its mandate along with risks to good governance and operations. Having identified the significant internal and external risks, the audit team prepares an entity risk profile that includes: a brief description of the risk and its impact on the entity s mandate, governance, and operations; a brief description of how internal and external risks to the entity affect the entity; 11

an assessment of the impact of the risk, as well as its likelihood (categorized as High, Medium, or Low); and a description of any previous audit work conducted by OAG Canada to address the risk identified. 4. Prepare the control profile The audit team prepares an entity control profile to document the assessment and consideration of the key controls within the entity. This profile helps to determine whether the controls are adequate for addressing the risks identified in the entity risk profile and for supporting the conclusion on the overall control environment. The entity control profile has two main components: Summary assessment of key controls: This provides an overview of the key controls that make up the overall control environment within the entity. Each control is assessed to determine its presence in a given area and its importance in responding to the risk identified in the entity risk profile. The audit team must also document the strengths and weaknesses of each of the controls. Consideration and assessment of key controls: A description of the key features of each control objective is documented, along with the strengths and weaknesses of the control objective and any factors that may mitigate the weaknesses identified. 5. Identify potential products and assign priorities Potential audit products for the next three to five years are identified and assigned priority using the following key procedures: Identify possible products: The audit products identified will depend on the focus of the Auditor General and will usually address multiple risks facing the entity. Care is exercised to limit the products to the risks that were given higher priority. Respond to entity risks: The proposed product is aligned to the risks that were identified. Assess relative priority: This procedure is undertaken after considering whether the product supports the Auditor General s focus areas and whether there has been previous audit coverage or other independent reviews of the subject matter. Document risks to the office: These are the risks of delivering (auditability) or not delivering (credibility) the proposed product. 12

6. Report to the Executive Committee The final step in the OPP process is to prepare and present a report to the Executive Committee of OAG Canada. The discussions during the presentation are kept at a high level, with attention only to key risks facing the entity that have not been mitigated. The report is checked by a quality reviewer, who provides feedback on its completeness. It is then presented to the Executive Committee. 13

Bibliography Bergin, Ron. Strategic Planning in the Office of the Auditor General of Canada, Presentation to the Canadian Comprehensive Audit Foundation Fellows, October 2009. Hopwood, Tom and Wiltshire, Collin. OAG Risk Strategies for the Next Decade. Discussion paper on use of risk concepts in OAG planning, auditing, and management, September 2001. Office of the Auditor General of Canada, Strategic Planning and Professional Practices. One-Pass Planning Guidance to the Entity Team, March 2002.. Update on our Strategic Plan Challenges, PowerPoint presentation, October 2003. Linked to the following document: http://notes.oag-bvg.gc.ca/intranet/intranet_menus.nsf/html/e_index.htm. Performance Audit Manual, June 2004.. Guidance on Preparing One-Pass Plans, September 2004. Available from: http://notes.oag-bvg.gc.ca/intranet/intranet_menus.nsf/html/e_index.htm. One Pass Plan for the Canada Revenue Agency, 2007.. One Pass Plan for the Government of Nunavut, 2007. Treasury Board of Canada Secretariat. Integrated Risk Management Framework, 2001. Available from: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12254. Integrated Risk Management Framework: A Report on Implementation Progress, March 2003. Available from: http://www.tbs-sct.gc.ca/rm-gr/irmf-cgir/2003-03- rprt01_e.asp 14