Australian National Audit Office. Report on Results of a Performance Audit of Contract Management Arrangements within the ANAO

Transcription

1 Australian National Audit Office Report on Results of a Performance Audit of Contract Management Arrangements within the ANAO

2 Commonwealth of Australia 2002 ISBN

3 10 December 2002 Dear Mr President Dear Mr Speaker In accordance with the provisions of Section 45 of the Auditor-General Act 1997, I submit to the Parliament my report on a performance audit of the Australian National Audit Office. The report presents my findings from a performance audit of the Australian National Audit Office s contract management arrangements. Yours sincerely Michael Coleman Independent Auditor Appointed under Section 41 of the Auditor-General Act 1997

4 Contents 1 1. Tender Evaluation process AASG 2. Selection of external contractors PASG On going contract management PASG On going contract management CMB Procedural guidance AASG and PASG Contract evaluation AASG and PASG Diversification of the contract source AASG Contract management risk corporate issue 25 1

5 Executive summary Introduction In accordance with Section 45 of the Auditor-General Act 1997, the independent auditor may at any time conduct a performance audit of the Agency and report in accordance with the requirements of the Act. The purpose of the performance audit is to consider the economy, efficiency and administrative effectiveness of core processes in place within the ANAO. This performance audit focused on the contract management arrangements in place within the Australian National Audit Office (ANAO). The review considered the contract management process in place within the Corporate Management Branch (CMB), Assurance Audit Services Group (AASG) and the Performance Audit Services Group (PASG). The core principles that underpin the contracting of services associated with ANAO activities are derived from the Commonwealth Procurement Guidelines and Best Practice Guidance issued in September This document was utilised by the ANAO to develop internal guidance on: contracting audits; engaging contract auditors; and engaging consultants In order to ensure the ANAO meets the objectives set out within the guidelines, a policy of open and public tendering has been adopted. To streamline its processes, the ANAO has established a number of panels of contractors and consultants with particular skills to meet its ongoing needs. Objective The objective of this performance audit was to perform an independent and systematic examination of the processes in place within the ANAO over the use of contractors to: undertake financial statement or performance audits under the direction of ANAO staff; undertake particular research or studies or a particular task associated with an audit under the direction of ANAO staff; and provide specialist services, including the assistance with the development and/or implementation of purpose-built IT systems and legal advice. Performance audit approach The approach to this performance audit included the following key steps: conduct of a series of meetings with key management involved in the contract management process; review of existing guidelines, procedure manuals and the ANAO s Better Practice Guide to Contract Management; 2

6 review of a sample of contract management files; identifying opportunities for improvement that will enhance processes within the ANAO; and report findings to management and seek responses to recommendations. My audit considered the following stages in the contract management lifecycle 1 : 1. Selecting the acquisition strategy 2. Developing and releasing the tender documentation 3. Evaluating the tender bids 4. Decision and implementation 5. Ongoing management 6. Evaluation and transition planning This audit has been performed in accordance with Australian Auditing Standards applicable to performance audits (AUS806 Performance Auditing) and accordingly included such tests and other procedures that I considered necessary in the circumstances. Conclusion My performance audit indicated that contract management processes in place within the ANAO are generally operating in an efficient and effective manner. Processes followed within all branches of the ANAO are closely aligned to better practice as set out in the ANAO s Better Practice Guide to Contract Management issued in February 2001, referred to hereafter as Better Practice. Through a review of the processes in place and a review of a sample of contracts my performance audit did identify a number of opportunities for improvement in the contract management process. Figure 1 summarises the extent to which processes supporting the contract management cycle are operating in an effective and efficient manner within the ANAO. My performance audit also considered the application of risk management principles to the contract management process. The ANAO s Better Practice Guide publication, Contract Management noted that the difference between a contract delivering benefits and one that does not can often be attributed to the way that the risks associated with the delivery of those services are managed. My review indicated that there is no consistent, formal assessment within either the AASG, PASG or CMB of the risks associated with the delivery of the service or the risk associated with the management of each contract. 1 The stages in the contract management lifecycle are taken from the ANAO s Better Practice Guide to Contract Management, February 2001, Table 1, p. 8. 3

7 Figure 1 Effectiveness and efficiency of processes supporting the contract management cycle Selecting the acquisition strategy Developing and releasing the tender documentation Evaluating the tender bids Decision and implementation Ongoing management Evaluation and transition planning AASG PASG CMB Key Processes in place are operating as designed in an effective and efficient manner and meet the requirements of the ANAO. The process is consistently followed within the group. Processes in place are not consistently operating as designed and there is some scope for improvements in process in order to align with best practice. 4

8 Contract management process Introduction The ANAO employs the services of contractors and consultants in essentially two ways. It contracts out audits, in whole or in part, to private sector organisations and contracts the provision of non-audit support services such as IT, publishing and legal services. It also contracts in staff to undertake audit and non audit work under the direction of ANAO staff, as well as consultants to undertake particular tasks which might be part of an audit or which may relate to non audit activities such as the development of computing software or training programs. During , the ANAO used approximately 110 contractors or consultants to provide a range of professional, independent and expert advice or services. These included arrangements with private sector organisations for the delivery of a range of support services, including IT, publishing and legal services. A total of $ million in contracts of greater than $10,000 (representing 200 contracts) were let or managed by the ANAO during The total amount of contracts greater than $10,000 let in was $7.023 million. Figure 2 outlines the split of contracts managed by each of the service groups. Figure 2 Active contracts managed by service groups Active Contracts by Service Group (%) 12% 24% 64% AASG CMB PASG The remainder of this report discusses arrangements in place within the ANAO for each stage of the contract management process. 5

9 Selecting the acquisition strategy This stage of the contract management cycle involves consideration of the activity to be undertaken and an examination of the options available to best satisfy the needs of the ANAO. In order to determine whether an activity should be delivered through a contractual arrangement consideration is given to the: Reason for the requirement; Alignment of the requirement with the ANAO s strategic objectives; Range of options available to meet the needs of the ANAO; and Achievement of a value for money outcome Assurance Audit Services Group On an annual basis a review is undertaken of the strategy in place for the use of the private sector in the conduct of financial statement audits. The objective of the review is to examine the use of the private sector in the conduct of financial statement audits and maximise the benefits to the ANAO in assisting it to achieve the key results of: Meeting client needs; Quality products and services; High performing people; and Good business management Current practice within the ANAO is to contract out the financial statement audits of government business entities (GBE) and specialised commercial industry clients such as those within the insurance industry 2. This has enabled the ANAO to focus internal resources on core government agencies. Prior to any decision to seek selective tenders for specific audit assignments an analysis is undertaken on the advantages and disadvantages of seeking the private sector to tender for those assignments. Within the AASG the strategy is to outsource some of the ANAO core business but train internal staff with the appropriate skills to allow these contracts to come back in house. Another strategic decision is to ensure the ANAO have IT literate auditors in place within the organisation and training workshops have been developed in order to increase IT skills within the organisation. 2 In such circumstances, the Auditor-General or delegated officers sign the audit opinions and are involved in the planning of, and significant meetings for, outsourced audits. 6

10 Performance Audit Services Group As part of the performance audit planning process the audit manager will develop an audit work plan which includes detail of the resources deemed necessary to complete the audit. During this stage the need for specialist assistance with the performance audit may be identified, at which point the audit manager will consult with the branch head and prepare a business case for the use of a consultant. Consultants can either be selected from the PASG panel, for the provision of generalist audit skills or outside the panel for specialist skills. Membership of the panel is reviewed every two years where interested parties are invited to outline their credentials and experience of performance audits within the public sector. A selection process then follows in which firms or individuals are chosen based on technical ability and value for money. Given the specialist nature of the work undertaken by the PASG the population of organisations or individuals who can provide the specialist services necessary is narrow. Corporate Management Branch The Corporate Management Branch follows the ANAO s acquisition strategy, which is based on procurement benchmarks, expressed in monetary terms, as points of reference to differentiate between the three broad ranges of complexity of supply. Three benchmarks are in place: Below $30,000 Contracts in this range would be for minor projects and would generally involve straight forward selection from the market place. $30,000 $100,000 Procurement methods for this level would include direct approach to known suppliers in the market for a minimum of three oral or written quotations, or more formal arrangements such as requests for tender. The choice of method and the extent of competition is a matter of judgement by the relevant official. Above $100,000 Where expenditure is likely to exceed $100,000, procurement methods include either one or a combination of requests for tender, requests for quotation, requests for proposals and invitations to register interest. My review of arrangements within the CMB focused on projects where expenditure was in excess of $100,000. For such projects a specification is designed outlining areas such as the main purpose and objectives of the contract, the specific tasks to be undertaken and the evaluation criteria to be used in assessing the bids from prospective contractors. My review focused on the two main contracts managed by the CMB, namely publishing services and the provision of IT services. The process behind selecting the acquisition strategy was consistent with best practice guidelines in both contracts. 7

11 Developing and releasing tender documentation The tender document is the prime source of information in relation to contract objectives and requirements. It details the roles and responsibilities of the contractor and the ANAO. The tender document is used to outline subjects such as the scope of the tender, selection criteria and the evaluation methodology. My review indicated that arrangements around the development and release of tender documentation are well managed throughout the ANAO. When providing a specification to prospective tenderers, a package consisting of a covering letter inviting tenders, a statement of requirement and a copy of the proposed contract is forwarded to prospective tenderers. For each tender released by the ANAO, an industry briefing is held to ensure that the ANAO receives quality responses to the request for tender as well as providing a level playing field for all prospective tenderers. It can also provide an opportunity for prospective tenderers to clarify any of the tender requirements and can assist them to scope the audit task or corporate requirement. The development and release of tenders within the ANAO is conducted in a manner designed to promote transparency, fair competition and provide best value for money. Given the majority of the contracts managed by the ANAO relate to the AASG, the emphasis of my review on the development and release of tender documentation was within this group. Assurance Audit Services Group Request for tender documentation in the AASG outlines the broad evaluation criteria groupings including categories such as: Compliance and financial viability this category requires tenderers to provide information on areas such as: compliance with the conditions of tender, appropriateness of insurance arrangements, writs or lawsuits pending, financial viability and conflicts of interest; Audit methodology, approach and management processes to be used this category requires tenderers to provide information on items such as an outline of the audit approach to be adopted, the reports to be delivered and the quality control policies and assurance procedures to be employed throughout the audit engagement; Qualifications and experience of the nominated professional staff tenderers are required to provide details of professional staff, the level of experience held by staff and other information including the percentage time partners and managers will devote to the tendered audit; Industry audit experience tenderers are required to demonstrate that they have the experience, commitment, capability and capacity to undertake audit services of the type identified in the request for tender; and Fee details Standard tender documentation provides guidance and sets out the ANAO s requirements allowing prospective tenderers to respond comprehensively to the request for tender. The tender documentation includes details of pre requisite conditions with which the tenderer is 8

12 expected to comply in submitting a tender response. In addition, a statement of requirement is included in the tender documentation setting out the overall requirement, services that are required, performance levels required, assistance provided by the ANAO and any other matter that is deemed critical to the performance of the contract. Performance Audit Services Group If the acquisition strategy to be followed requires a consultant to be selected from outside the panel a request for tender will be developed. The process followed in developing and releasing the tender documentation is in line with that used in the AASG. Generally, the value of individual PASG contracts is small in comparison to contracts issued by AASG. Corporate Management Branch At the time of my performance audit the CMB were in the process of announcing the tender for the provision of IT services to the ANAO. It appeared to me that arrangements in place for developing and releasing the tender documentation for this contract promoted openness through the holding of industry briefings in an effort to ensure all prospective tenderers had access to relevant information. Evaluating the tender bids The evaluation of offers is concerned with the assessment, comparison, elimination and selection of offers. This stage of the contract management process is used to determine each tenderer s technical, financial and managerial capacity to supply and meet all contractual obligations if successful. It is important that offers are evaluated in accordance with the tender process, as set out in the tender document. My review indicated that the process for evaluating tender bids was consistent throughout the three divisions. A tender evaluation team is appointed to undertake the tender evaluation. The membership of this team is generally consistent with the team established to develop the tender documentation and are therefore well placed to assess the tenders against the nominated criteria. The ANAO adopt a two envelope system where the tender response is provided in two separately sealed envelopes: the first containing the pricing information and the second containing the remainder of the tender response. In using this approach the evaluation of technical worth is undertaken separately from that on price and the two elements are brought together at the end to determine which offer represents best value for money to the ANAO. Within the AASG, the composition of the evaluation team may include an independent person such as a representative of the audit client depending on the size and scale of the tender. As part of my review I undertook a detailed review of the contract management processes in place with regard to the AASG contracts in respect of the financial statement audits of the largest outsourced audit contract, Telstra and another selected at random, namely the Defence Housing Authority. 9

13 Telstra My review of the evaluation of the tender bids in respect of the Telstra contract indicated that the process followed was in line with ANAO procedural guidelines and included the involvement of an independent probity adviser. The role of the probity adviser was restricted to the review of the procedural integrity of the tender process. The probity adviser had no involvement in the evaluation of proposals. A document setting out the evaluation and completion processes to be observed by the evaluation committee was developed and agreed between the ANAO and Telstra. The first stage of the evaluation was to review the proposals received against compliance with the confirmation requirement set out in the tender documentation. Tenderers meeting the confirmation requirement were invited to present to the evaluation committee in order to amplify their proposal and allow the committee to seek clarification on the respective proposals. Following the presentation process, committee members individually considered the merits of each proposal and analysed in more detail each submission against the defined criteria. The committee and probity adviser then met to consider the merits of each proposal against criteria. Together with the weightings, ratings out of ten were applied to the individual information requirement aspects of each criterion in order to ensure a scoring process was adopted. Defence Housing Authority My review of the evaluation of the tenders submitted in respect of the financial audit contract for the Defence Housing Authority indicated that the majority of processes followed were in line with documented guidelines. However, I did note that the evaluation process outlined within the tender documentation was not followed exactly as set out. The tender document indicated that as a part of the first stage of the elimination process, the tender evaluation team was to examine each submission s claim against set criteria and apply a weighted scoring methodology to rank each tender. In the event, only two of a possible four tenderers responded. Because of the small response, the evaluation team truncated the preliminary evaluation stage and proceeded straight to the step of minuting that both tenderers had met the selection criteria. There was no formal documentation of how the tenderers had satisfied the elimination criteria that, in accordance with the tender document, were intended to involve a weighted scoring methodology. This approach allowed the tender evaluation team to proceed to the next step in the evaluation process that was set out in the tender document. The final decision was made after considering the remaining steps that were noted in the tender documents and applying the ANAO s usual approach of determining value for money. As a matter of good governance we believe that there should have been a formal note to document how the tenderers had met the selection criteria, even though there were only two tenderers. Recommendation 1 In evaluating proposals, the ANAO should ensure that any evaluation process outlined within the request for tender document is followed and that sufficient evidence is held on file to demonstrate that the process as set out in the tender documentation has been followed. 10

14 Decision and implementation This stage in the contract management cycle involves recommendation of the preferred tenderer as agreed by the evaluation panel. This stage also involves the formation of the contract, issue of the acceptance letter to the successful tender and notification and debriefing to the unsuccessful tenderers. Processes to be followed during this stage of the contract management cycle are common throughout the three divisions. Current practice is to advise the successful tenderer and unsuccessful tenderers of the decision by letter. These letters are issued at the same time and the unsuccessful tenderers are offered the opportunity of a debriefing session. The draft contract provided as part of the request for tender process forms the contract and provides the basis for any contract negotiations. The contract will be sent to the successful tenderer for signing prior to signing by the ANAO. A suite of standard contracts has been developed for use by ANAO staff in setting up most of its consultancies. These can be adapted readily by ANAO staff to suit individual circumstances. During 2001, the ANAO undertook an exercise to review and streamline its existing standard contracts and used a legal provider on the relevant ANAO panel to review the existing set of standard contracts. That work has resulted in a revised set of contracts designed to be more user friendly and easier for ANAO staff to follow. ANAO staff can also access a panel of legal experts who can provide specialist legal advice, for both contract issues and other legal services. If a contract is required to be varied in any way, the project manager must submit a business case to the relevant Group Executive Director seeking approval. Once the approval has been received, Practice Management will prepare the variation and forward this to the relevant signing officer for signature and on forwarding to the contract auditor. My performance audit of the processes in place in relation to the decision and implementation of contracts indicated that these were well considered and operating effectively. However, there is one area where current processes could be improved. During the PASG audit planning process it is common for the need to employ a consultant to be identified. My performance audit indicated that while there was appropriate authorisation within the audit work plan to employ external assistance, the specific reason behind the need to employ a specific consultant was not always explicit in the related documentation. Good governance suggests that the reasoning behind the selection of a specific contractor be outlined within documentation supporting the audit work plan. This will ensure there is transparency in the decision making process and will protect the ANAO against any accusations of favourable treatment to a particular consultant. Recommendation 2 To ensure there is transparency in the selection of external consultants involved in performance audits, documentation supporting the audit work plan should explicitly outline the reasoning behind the decision to select a particular consultant. 11

15 Ongoing management of contracts This stage of the contract management cycle covers the life of the contract from the commencement to its termination date. The principal objectives of ongoing contract management are to: Develop service level agreements; Develop a performance management system; Manage ongoing, day to day issues; and Deal with the possible effects of dissatisfaction with service delivery. With regard to the AASG and the PASG, the work of the ANAO must be carried out against the requirement for reporting to Ministers and to Parliament. The reliance placed by Parliament on the objectivity, accuracy and balance of reports made by the Auditor-General or under his authority is of fundamental importance to the further steps taken by Parliament in its scrutiny of public administration. Effective parliamentary follow-up cannot take place if the audit reports on which action is based are defective in terms of fact or the findings expressed in them. It follows that the work of all professional audit staff has to be of the highest possible standard so that adequate and reliable material is produced in support of a recommendation that a report be signed expressing findings, opinions and recommendations. The Deputy Auditor-General and his staff are required to monitor audits and review results of audits contracted out to agents of the Auditor-General as they are responsible for quality assurance within the range of portfolios allocated to their group. My review considered the processes in place within the ANAO over the monitoring of financial statement and performance audits contracted out to third parties. Assurance Audit Services Group The ANAO Policy Manual relating to the Quality Assurance reviews of project managed financial statement audits indicates that each year, quality assurance reviews will be conducted on a sample of audits under project management arrangements. These reviews include: A review of the firm's normal quality assurance and control procedures, as set out in relevant manuals, for adequacy against professional requirements; and A review of actual audits to ensure that the firm's quality assurance policies and procedures were in fact applied to those audits, and to ensure that the Audit Project Manager's involvement with the audit was appropriate given the circumstances (e.g. the level of risk) of the audit. The reviews do not involve consideration of the basic audit work. They address management, control, supervision and assurance matters related to the audit. Review of the audit work itself is the responsibility of the contractor and Audit Project Manager. The sample of audits are selected on the following basis: Each major audit (that is, an audit with an audit fee in excess of $100,000 pa) will be reviewed at least once during the contract; 12

16 Each firm providing project managed services to the ANAO will be subject to review at least every second year. Where issues are identified in relation to a specific audit, these matters are followed up with the respective project manager during the subsequent year's review. Performance Audit Services Group Unlike the processes in the AASG, the quality assurance reviews on performance audits do not specifically target contract managed jobs. This reflects the fact that most PASG contracts relate to individuals or the provision of specific services. Accordingly, if the performance audit selected for a quality review used contractors to conduct the work, then the quality assurance process considers the management of the resources, but the quality of the contractor s performance is not a specific focus of the review. My performance audit indicated that the process for monitoring the performance of contractors used on performance audits was not as formal or detailed as that undertaken through the AASG. A review is undertaken of the contractors work as part of the performance audit review process but there is currently no formal process to utilise information gathered through this review process in evaluating contractor performance overall. While there is a need to ensure the review undertaken is commensurate with the risks involved in the performance audit, in my opinion greater emphasis should be placed on using the information gathered during this review to evaluate the performance of individual contractors used in the performance audit process. Recommendation 3 As part of the review process within the Performance Audit Services Group, consideration should be given to ensuring there is a formal mechanism for considering any issues identified during the review in relation to the performance of individual contractors in the evaluation process. Corporate Management Branch At the time of my performance audit the two largest contracts within the CMB related to printing services and IT services. A contract manager is in place for each of these contracts and their responsibility is to ensure risks from both external and internal sources are managed to ensure effective delivery of the contracted service. As the printing contract was awarded in December 2001, I focused my attention on the transition between providers and the on going management of this contract. One of the key elements of a successful contract relationship is good communication between the purchaser and provider. In developing a successful relationship the development of a good rapport between the relevant parties is important as is the need to ensure adequate time is taken to obtain the facts and find a mutually agreed solution to conflicts or performance issues as they arise. Through my work I became aware of a number of issues identified during the transition stage of the contract with audit staff raising several complaints about the quality of work produced by the publisher/printer. A review of the contract monitoring file indicated that a number of meetings had been held between ANAO representatives and the contractor in an attempt to resolve the issue. Management informed me that many of these meetings were informal and dealt with relatively minor issues related to the transition to the new arrangements. 13

17 Correspondence held on file indicated that subsequent to these meetings the contractor had provided assurances to resolve the issues identified. Although the contract file includes detail of all significant actions undertaken by the ANAO and contractor to resolve quality issues, minutes were only available for one of the several liaison meetings that had been held between the contract being awarded in December 2001 and October As a matter of good record keeping, I believe it would have been desirable to prepare short notes of the transition meetings. Recommendation 4 All formal regular liaison meetings should be minuted in order to maintain a record of discussions between the ANAO and printing contractor. Desirably, short notes of other informal operational meetings should also be maintained. This will ensure a record is maintained of significant issues arising during the term of the contract. Evaluation and transition planning This stage of the contract management cycle deals with processes involved in the: Evaluation of the contracted activity against tender requirements; Assessing effectiveness through feedback from clients and other stakeholders; and Consideration of new approaches at the completion of the current contract ANAO management give consideration to the evaluation of contracts and transition planning as contracts approach the end of their tenure. This typically involves the contract team meeting and considering whether the contract should be put out for tender. Any such decision will be based on the ANAO s strategic direction, relevant procurement guidelines and an evaluation of the contractor. The evaluation of contractors has not been formalised within the ANAO s guidelines for contracting audits, engaging contract audits or engaging consultants. As such there is a risk that an inconsistent approach to contract evaluation and transition planning within the AASG and CMB. Recommendation 5 To improve the effectiveness of contract evaluation, consideration should be given to updating procedural guidelines with detail of the processes to be followed in evaluating the performance of contracted auditors. These guidelines should also outline the transition planning process. Client surveys are undertaken by an independent consultant to determine the level of satisfaction with the services provided by the ANAO. Clients complete a questionnaire assessing ANAO performance across key audit areas. An interview is also held with the Agency to confirm the results of the survey and incorporate any additional feedback. Surveys are presented to Executive Directors for comment and to incorporate proposed actions and lessons learned where unfavourable results are received. The results are then issued to the Deputy Auditor-General and Auditor-General and communicated to the audit team. The results of client surveys form a component of the ANAO s key performance 14

18 targets. Where financial statement audits are contracted out, these surveys provide a basis for evaluation of the contractor s performance for a client perspective. Current practice within the ANAO is for client teams to consider the performance of contractors in isolation rather than undertaking an exercise in order to evaluate the performance of contractors, identify areas for improvement in the future and share examples of better practice. Recommendation 6 Given the same contractors are often used in a number of ANAO audits, consideration should be given to developing an appropriate mechanism for evaluations about the performance of contractors to be shared by client teams in order to highlight areas where contractor performance could be improved. My review indicated that until the demise of Arthur Anderson, that firm and Ernst & Young were the principal contractors for outsourced financial statement audits let by the ANAO. Ernst & Young has substantially acquired the business of Arthur Anderson and currently 83% of the value of AASG s active contracts are with the enlarged Ernst &Young. Following the demise of Arthur Anderson, an exercise was undertaken by the ANAO s senior management team in conjunction with legal advisors to consider the novation of contracts to Ernst & Young. While this exercise considered, inter alia, potential conflicts of interest, the decision to novate the contracts to Ernst & Young was not formally documented within supporting paperwork. Management informed me that the decision to novate the audit contracts rather than re-tender them was heavily influenced by the need to take speedy action to ensure the June 2002 audit program could be completed on time. This decision was not explicitly documented, notwithstanding that the ANAO sought and documented legal advice in relation to the issues that required consideration as part of the demise of Arthur Andersen. The contracts that have been novated to Ernst & Young have different review dates. In evaluating whether these novated contracts should be renewed, management should assess the appropriateness of holding such a large percentage of outsourced financial statement audits with one firm. Recommendation 7 Currently the ANAO has a significant proportion of its contracted financial statements audits with one firm. This arose from the demise of one major firm. When these contracts are due for renewal, the ANAO should consider the ongoing appropriateness of such a high concentration of its contracts being with one firm. Risk management and contracts The ANAO s Better Practice Guide publication, Contract Management notes that organisations may apply the principles of risk management to core business processes, but often do not effectively apply those same principles to contract management. Within both the AASG and PASG an assessment is undertaken of the risk associated with each audit to be undertaken through the ANAO. This risk assessment considers the risk associated with the audit but does not extend to consideration of the risks associated with 15

19 contracting out an audit to a third party. No assessment is undertaken of the contract management risk and the consequence and likelihood of such risks crystallising. While acknowledging that the publishing function had been outsourced for a number of years my review of arrangements around the publishing contract indicated that no formal assessment of the contract risk associated with the outsourcing of publishing services was undertaken before the contract was let. Through discussion with relevant staff I am aware that the risks associated with the contract were implicitly considered but there was no documented evidence to confirm this within the contract file. Recommendation 8 In line with better practice, consideration should be given in the contract management process throughout the ANAO to the formal identification and analysis of the risks associated with contract delivery that could have a detrimental impact on the ANAO. 16

20 Opportunities for improvement Implementation priority Low 1. Tender evaluation process AASG Page 18 Low 2. Selection of external contractors PASG Page 19 Medium 3. On going contract management PASG Page 20 Low 4. On going contract management CMB Page 21 Low 5. Procedural guidance AASG and PASG Page 22 Low 6. Contract evaluation AASG and PASG Page 23 Medium 7. Diversification of the contract source AASG Page 24 Medium 8. Contract management risk corporate issue Page 25 Implementation Priority High Medium Low Action Immediate action required by management to ensure that the contract management process is appropriately aligned with requirements of the ANAO Requires the attention of management and an agreed program for prompt resolution to ensure that the contract management process is appropriately aligned with requirements of the ANAO Attention is warranted by management to ensure that the contract management process is appropriately aligned with requirements of the ANAO 17

21 1. Tender evaluation process AASG Low Findings The current tender evaluation process is to assess each submission against set criteria and use a weighted scoring methodology to rank each tender. My review of the evaluation of the tenders submitted in respect of the financial audit contract for the Defence Housing Authority indicated that the majority of processes followed were in line with documented guidelines. However, I did note that the evaluation process outlined within the tender documentation was not followed exactly as set out. The tender document indicated that as a part of the first stage of the elimination process, the tender evaluation team was to examine each submission s claim against set criteria and apply a weighted scoring methodology to rank each tender. In the event, only two of a possible four tenderers responded. Because of the small response, the evaluation team truncated the preliminary evaluation stage and proceeded straight to the step of minuting that both tenderers had met the selection criteria. There was no formal documentation of how the tenderers had satisfied the elimination criteria that, in accordance with the tender document, was intended to involve a weighted scoring methodology. This approach allowed the tender evaluation team to proceed to the next step in the evaluation process that was set out in the tender document. The final decision was made after considering the remaining steps that were noted in the tender documents and applying the ANAO s usual approach of determining value for money. As a matter of good governance we believe that there should have been a formal note to document how the tenderers had met the selection criteria, even though there were only two tenderers. Key actions required: In evaluating proposals, the ANAO should ensure that any evaluation process outlined within the request for tender document is followed and that sufficient evidence is held on file to demonstrate that the process as set out in the tender documentation has been followed. ANAO comment: Agreed: The ANAO s guidance material will be amended to emphasise that the evaluation process outlined in the request for tender is followed in the evaluation of tenders and that documentation is maintained to evidence this. The evaluation process outlined in the ANAO s standard RFT will also be amended to indicate that the weighting of selection criteria is an option that should be considered on a case by case basis. 18

22 2. Selection of external contractors PASG Low Findings During the audit planning process it is common for the need to employ a consultant to be identified. My review indicated that while there was appropriate authorisation within the audit work plan to employ external assistance, the specific reason behind the need to employ a specific consultant was not always explicit in the supporting documentation. Good governance suggests that the reasoning behind the selection of a specific contractor be outlined within documentation supporting the audit work plan. This will ensure there is transparency in the decision making process and will protect the ANAO against any accusations of favourable treatment to a particular consultant. Key actions required: To ensure there is transparency in the selection of external consultants involved in performance audits, documentation supporting the audit work plan should explicitly outline the reasoning behind the decision to select a particular consultant. ANAO comment: Agreed: Action has been taken to ensure that the basis of selection of contractors is adequately documented. 19

23 3. On going contract management PASG Medium Findings Unlike the processes in the AASG, the quality assurance reviews on performance audits do not specifically target contract managed jobs. This reflects the fact that most PASG contracts relate to individuals or the provision of specific services. Accordingly, if the performance audit selected for a quality review used contractors to conduct the work, then the quality assurance process considers the management of the resources, but the quality of the contractor s performance is not a specific focus of the review. My performance audit indicated that the process for monitoring the performance of contractors used on performance audits was not as formal or detailed as that undertaken through the AASG. A review is undertaken of the contractors work as part of the performance audit review process but there is currently no formal process to utilise information gathered through this review process in evaluating contractor performance overall. While there is a need to ensure the review undertaken is commensurate with the risks involved in the performance audit, in my opinion greater emphasis should be placed on using the information gathered during this review to evaluate the performance of individual contractors used in the performance audit process. Key actions required: As part of the review process within the Performance Audit Services Group, consideration should be given to ensuring there is a formal mechanism for considering any issues identified during the review in relation to the performance of individual contractors in the evaluation process. ANAO comment: Agreed: Review processes will be revised and strengthened to ensure they specifically address the performance of individual contractors. 20

24 4. On going contract management CMB Low Findings Through my work I became aware of a number of issues identified during the transition stage of the contract with audit staff raising several complaints about the quality of work produced by the publisher/printer. A review of the contract monitoring file indicated that a number of meetings had been held between ANAO representatives and the contractor in an attempt to resolve the issue. Management informed me that many of these meetings were informal and dealt with relatively minor issues related to the transition to the new arrangements. Correspondence held on file indicated that subsequent to these meetings the contractor had provided assurances to resolve the issues identified. Although the contract file includes detail of all significant actions undertaken by the ANAO and contractor to resolve quality issues, minutes were only available for one of the several liaison meetings that had been held between the contract being awarded in December 2001 and October As a matter of good record keeping, I believe it would have been desirable to prepare short notes of the transition meetings. In not maintaining documentation to support contractual discussions, there is a risk that evidence is unavailable to confirm action items and respective responsibilities. Key actions required: All formal regular liaison meetings should be minuted in order to maintain a record of discussions between the ANAO and printing contractor. Desirably, short notes of other informal operational meetings should also be maintained. This will ensure a record is maintained of significant issues arising during the term of the contract. ANAO comment: Agreed: All future formal liaison meetings will be minuted. An appropriate record of other meetings held will also be maintained. 21

25 5. Procedural guidance AASG and PASG Low Findings The current contract evaluation process is based on the ANAO s strategic direction, relevant procurement guidelines and an evaluation of the contractor. The evaluation of contractors has not been formalised within the ANAO s guidelines for contracting audits, engaging contract audits or engaging consultants. As such there is a risk that an inconsistent approach to contract evaluation and transition planning within the AASG and CMB. Key actions required: To improve the effectiveness of contract evaluation, consideration should be given to updating procedural guidelines with detail of the processes to be followed in evaluating the performance of contracted auditors. These guidelines should also outline the transition planning process. ANAO comment: Agreed: The ANAO will update its procedural guidance to include further guidance on the evaluation of the performance of contract auditors and the need for appropriate transition arrangements to be put in place. 22

26 6. Contract evaluation AASG and PASG Low Findings As part of the evaluation process, client surveys are undertaken by an independent consultant to determine the level of satisfaction with the services provided by the ANAO. Surveys are presented to Executive Directors for comment and to incorporate proposed actions and lessons learned where unfavourable results are received. The results are then issued to the Deputy Auditor-General and Auditor-General and communicated to the audit team. The results of client surveys form a component of the ANAO s key performance targets. Where financial statement audits are contracted out, these surveys provide a basis for evaluation of the contractor s performance for a client perspective. Current practice within the ANAO is for client teams to consider the performance of contractors in isolation rather than undertaking an exercise in order to evaluate the performance of contractors, identify areas for improvement in the future and share examples of better practice. Key actions required: Given the same contractors are often used in a number of ANAO audits, consideration should be given to developing an appropriate mechanism for evaluations about the performance of contractors to be shared by client teams in order to highlight areas where contractor performance could be improved. ANAO comment: Agreed: Existing arrangements will be strengthened to provide for the sharing of information on the evaluation of contractor performance. 23

27 7. Diversification of the contract source AASG Medium Findings Current practice within the ANAO is to contract out government business entities (GBE) and specialised commercial industry clients such as those within the insurance industry. This has enabled the ANAO to focus on core government agencies. Prior to any decision to seek selective tenders for specific audit assignments an analysis is undertaken on the advantages and disadvantages of seeking the private sector to tender for those assignments. My review indicated that until the demise of Arthur Anderson, that firm and Ernst & Young were the principle contractors for outsourced financial statement audits let by the ANAO. Ernst & Young has substantially acquired the business of Arthur Anderson and currently 83% of the value of AASG s active contracts are with the enlarged Ernst &Young. Following the demise of Arthur Anderson, an exercise was undertaken by the ANAO s senior management team in conjunction with legal advisors to consider the novation of contracts to Ernst & Young. While this exercise considered, inter alia, potential conflicts of interest, the decision to novate the contracts to Ernst & Young was not formally documented within supporting paperwork. Management have informed me that the decision to novate the audit contracts rather than re-tender them was heavily influenced by the need to take speedy action to ensure the June 2002 audit program could be completed on time. It would have been difficult to re-tender the contracts in the available time. This decision point was not explicitly documented, notwithstanding that the ANAO sought and documented legal advice in relation to the issues that required consideration as part of the demise of Arthur Andersen. The contracts that have been novated to Ernst & Young have different review dates. In evaluating whether these novated contracts should be reviewed, management should assess the appropriateness of holding such a large percentage of outsourced financial statement audits with one firm. Key actions required: Currently the ANAO has a significant proportion of its contracted financial statements audits with one firm. This arose from the demise of one major firm. When these contracts are due for renewal, the ANAO should consider the ongoing appropriateness of such a high concentration of its contracts being with one firm. ANAO comment: Agreed: While the ANAO appreciates the benefits of diversification, the current high concentration of contracts with one provider is the result of the recent merger of our two leading providers. Individual contracts are awarded on the basis of best value for money and have resulted in varying proportions of outsourced work to major accounting firms over many years. The ANAO will continue to select contract auditors on a value for money basis having regard to the contract risks, including any risks associated with a concentration of audits with a small number of firms. 24