Audit of Entrepreneurship and Business Skills Development (BDP, WBI, YEDI) ACOA Internal Audit Directorate Final Report December 2007
TABLE OF CONTENTS Assurance Statement... 5 Executive Summary... 6 Background... 7 Audit Risk... 8 Audit Objectives... 9 Audit Scope and Methodology... 9 Audit Results... 11 Objective 1:... 11 Objective 2:... 14 Objective 3:... 14 Objective 4:... 15
ASSURANCE STATEMENT President, Atlantic Canada Opportunities Agency We have completed the Audit of Entrepreneurship and Business Skills Development, which included the Women in Business Initiative (WBI), the Young Entrepreneur Development Initiative (YEDI), and certain elements of the Business Development Program (BDP). The overall objective of this audit was to provide assurance that the Agency has exercised due diligence in the delivery of program activity supporting Entrepreneurship and Business Skills Development (EBSD) and has complied with established policies and guidelines, as well as the Treasury Board Policy on Transfer Payments. The examination was conducted during the period of May 2007 through July 2007, and covered contributions approved during the three-year period that commenced April 1, 2004 and ended March 31, 2007. The audit consisted of an examination of project files in respect of approved contributions from the Agency s four regional offices, Head Office and Enterprise Cape Breton Corporation (ECBC). Selected projects were examined in detail to evaluate compliance with Agency guidelines through all project phases, including assessment, disbursement, administration and monitoring. We concluded that, on an overall basis, ACOA exercises due diligence in assessing EBSD-related applications and in disbursing program funds to recipients. We further concluded that project approvals and amendments are in accordance with established delegated authorities, contracts contain provisions as required by program policies and guidelines, and monitoring activities are adequate to meet established requirements. In our judgment, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions were based on a comparison of the situations as they existed at the time against the audit criteria and in consideration of the results of the compliance testing. The conclusions are only applicable in respect of the programs audited. Chief Audit Executive Internal Audit Direcorate Page 5
EXECUTIVE SUMMARY The Audit of Entrepreneurship and Business Skills Development focused on the Entrepreneurship and Business Skills Development (EBSD) priority associated with three programs or initiatives: the Business Development Program (BDP), the Women in Business Initiative (WBI) and the Young Entrepreneur Development Initiative (YEDI). The program and initiatives provide financial assistance to not-for-profit organizations, which in turn undertake activities that help foster entrepreneurship and business skills development among small and medium-sized enterprises (SMEs), and are delivered through the ACOA regional offices, Head Office and ECBC, with responsibility for development and maintenance of policy and procedures resting with program management at Head Office in Moncton. The Audit of Entrepreneurship and Business Skills Development respecting BDP, WBI and YEDI Entrepreneurship-related projects for the period April 2004 to March 2007 was undertaken with the objective of providing assurance that ACOA exercised due diligence in the delivery of the program/initiatives and followed Treasury Board Policy on Transfer Payments. From the evidence gathered, we conclude that almost all projects met eligibility requirements. The results also indicated that these projects were properly assessed against evaluation criteria, and that information critical to project decisions was consistently and accurately documented. Recommendations were made to improve certain elements of the assessment process to ensure consistent documentation of applicant and project eligibility. The payments system was examined and solid management processes were found to be in place. The audit examined project monitoring and evaluation of project benefits, and noted some areas in which we believe performance could be improved. Recommendations were made to improve the monitoring process and emphasize the importance of formally documenting project achievements. Some recommendations in this audit report were also noted in an earlier audit of the BDP. Because both audits examined program activity for a similar period, management action in response to the BDP audit would not have been evident at the time the audit fieldwork was completed for this engagement. Accordingly, this audit could not take into account corrective measures implemented in response to the BDP audit. Based on the results of the audit, the Internal Audit Directorate has issued a Statement of Assurance that, on an overall basis, the Agency exercised due diligence in the delivery of the Entrepreneurship and Business Skills Development elements of the BDP, WBI and YEDI. Page 6 Internal Audit Directorate
BACKGROUND ACOA invests in EBSD initiatives to encourage more Atlantic Canadians to develop the motivation and skills required to successfully start, sustain and grow a business. The Audit of Entrepreneurship and Business Skills Development, approved by ACOA s Review Committee (currently Audit Committee), is an assurance engagement that was included in the Agency s Audit Plan for the 2006-2007 fiscal year. As a component of the Agency s Program Activity Architecture (PAA), the Entrepreneurship and Business Skills Development activity encompasses a number of contribution programs, including the WBI and YEDI (under the Atlantic Investment Partnership), and certain elements of the BDP. The objectives of each program/initiative are as follows: The BDP Entrepreneurship and Business Skills Development element is intended to stimulate the development of existing and potential entrepreneurs by increasing their awareness of entrepreneurial opportunities and by enhancing business management skills. The key EBSD components of the BDP are youth entrepreneurship, promotion and awareness, entrepreneurship education and networking. The goal of the WBI is to improve the growth and competitiveness of women-owned businesses in Atlantic Canada. More specifically, the objectives of the WBI are to strengthen the management capabilities and business development skills of women-owned businesses, improve access to capital and business support services for women-owned businesses, and increase the involvement of women-owned businesses in exporting and innovation activities. The overall objective of YEDI is to enable more young Atlantic Canadians to develop the motivation and skills required to successfully start, sustain or grow a business. YEDI s three-fold approach to meeting its overall objective is to: provide aspiring young entrepreneurs with opportunities to develop practical entrepreneurship and business skills; improve the ability of existing young entrepreneurs to succeed and grow by facilitating mentoring, on-line support, diagnostic services and general business skills development; and facilitate community-level participation that supports young entrepreneurs by engaging targeted communities or regions in the preparation of development plans and activities. Internal Audit Direcorate Page 7
During the period covered by the audit (April 2004 March 2007), the Agency approved contributions for Entrepreneurship and Business Skills Developmentrelated projects under the BDP, WBI and YEDI as follows. Region BDP WBI YEDI Totals General + EBSD CAS O&M CAS G&C ($) ($) ($) ($) ($) ECBC 592,890 112,485 14,288 479,343 1,199,006 NB 3,071,795 1,798,337 0 2,293,857 7,163,989 NL 4,521,423 791,351 441,066 2,303,802 8,057,642 NS 7,941,777 1,779,253 498,374 2,254,308 12,473,712 PE 1,330,822 1,483,930 0 428,072 3,242,824 HO 2,215,443 95,000 0 1,912,447 4,222,890 TOTALS 19,674,150 6,060,356 953,728 9,671,829 36,360,063 *Note: These figures do not include multi-year projects approved prior to April 2004 that were in progress during the audit period. AUDIT RISK The Audit of Entrepreneurship and Business Skills Development is included in the Audit Plan for 2006-2007 as approved by Review Committee (currently Audit Committee). Engagement risk is determined through the use of the Audit Universe Risk Analysis (AURA). The AURA specifies the engagement risk by considering six risk criteria: Level of Funding Other Audits Performed 3 rd Party Control and Delivery Agency Internal Controls Program Sensitivity Agency Strategic Direction Through the AURA, the engagement risk was determined to be high as a result of no prior auditing and the fact that Entrepreneurship and Business Skills Development is a component of the Agency s primary strategic outcomes. Page 8 Internal Audit Directorate
AUDIT OBJECTIVES The overall objective of the audit was to provide assurance that the Agency has exercised due diligence in the delivery of EBSD-related program / initiative activity related to the BDP, WBI and YEDI, and has complied with established policies and guidelines, as well as the Treasury Board Policy on Transfer Payments. To provide assurance, a number of sub-objectives were established. ACOA exercises due diligence in assessing eligibility of applications for contributions. Project approvals and amendments are in accordance with established delegated authorities and related contracts contain provisions as required by program policies and guidelines. The Agency exercises due diligence in disbursing program funds to recipients. Reporting and monitoring activities are adequate to meet the requirements established for each project during the assessment and approval process. AUDIT SCOPE AND METHODOLOGY The engagement was included in the 2006-2007 ACOA Audit Plan, approved by ACOA s Review Committee (currently Audit Committee). The audit was carried out during the period May 2007 through July 2007. The audit examined contributions approved during the period April 2004 through March 2007. The audit consisted of an examination of project files relating to approved contributions from the Agency s four regional offices, Head Office and ECBC. Selected projects were examined in detail in order to evaluate compliance with Agency guidelines through all project phases, including assessment, administration, disbursement and reporting/monitoring. Internal Audit Direcorate Page 9
The number of EBSD projects selected for examination, as well as the total related contributions, is as follows. Region BDP WBI YEDI Total $ Assistance EBSD General + CAS G&C CAS O&M Selected Projects ECBC 2 2 0 6 814,125 NB 6 6 0 6 3,594,739 NL 7 6 3 7 4,466,198 NS 5 5 3 5 4,363,138 PE 6 3 0 3 2,421,830 HO 4 1 0 5 2,442,530 TOTALS 30 23 6 32 18,086,724 A visit to each regional office was made to: conduct interviews with management to obtain an understanding of systems and processes in place; and complete a detailed examination of project files in order to ensure compliance with Terms and Conditions of the initiatives, program guidelines and the Treasury Board Policy on Transfer Payments. Page 10 Internal Audit Directorate
AUDIT RESULTS OBJECTIVE 1 Provide assurance that ACOA exercises due diligence in assessing eligibility of applications for contributions. Criterion 1 - Applicants, projects and project costs are satisfactorily screened for eligibility prior to approval. Eligibility According to Agency guidelines, eligibility as it relates to entrepreneurship and business skills development encompasses primarily non-commercial organizations that undertake activities to foster EBSD and that adhere to program eligibility requirements. Eligible costs are those considered reasonable and necessary to carry out the project and adhere to program guidelines. Further, each project must support at least one of the Agency s strategic outcomes. For those projects associated with WBI and YEDI, we are satisfied that selected projects met the eligibility requirements established for the initiatives. We found a clear link between the projects and the strategic priority. For the BDP projects reviewed, eligibility was apparent; however, the link to the EBSD strategic priority was not always evident. Such projects were specifically attributable to contributions designated under the EBSD strategic priority for which the EBSD group has no management responsibility. Recommendation: Where the applicant or activity does not have a direct or apparent link to the EBSD strategic priority, the rationale for the determination of project eligibility should be specifically documented. Criterion 2 - All information critical to project decisions is documented on Project Summary Forms (PSFs) and/or included in project files. The PSF is the primary tool used to capture key project information. Based on our review, it is clear that project files, including PSFs, contain the required information elements needed to support funding decisions. In almost all cases, details of project costs, funding sources, required financial analyses and program information were included in PSFs during the evaluation process; however, we noted certain areas where we believe documentation could be improved. Internal Audit Direcorate Page 11
Project Costs At the time of project assessment, the Account Manager s review of proposed costs against project scope and eligibility criteria, and the conclusions on acceptability of costs determine final cost categories for inclusion in the contribution agreement. It is important that project files document the extent to which project costs and other assumptions have been challenged by the Account Manager. Based on our audit, the extent of the challenge performed was sometimes unclear. Recommendation: Evidence that proposed costs and other assumptions have been challenged by the Account Manager, and the basis on which costs have been determined, should be documented in the project file. Other Funding To assess overall viability and ensure compliance with established stacking limits, assistance from all other sources (i.e. federal, provincial, municipal governments, the applicant or corporate sponsors) must be considered during project assessment, and should be clearly stated on the PSF and Statement of Work. We noted a number of instances, however, in which it was not clear where the other funding originated, its intended purpose and, the department and program from which any other funds were sourced. Recommendations: For project funding that is in addition to ACOA s contribution, the source, amount of funding and purpose should be clearly described on both the PSF and Statement of Work. Criterion 3 - Projects are assessed against required criteria in respect of incrementality/need, commercial viability and net economic benefit. Program guidelines provide assessment considerations to facilitate project evaluation. In almost all cases, we found that applications for assistance under the EBSD were evaluated using the assessment criteria of incrementality, viability and net economic benefit. We did note, however, opportunities for improvement with respect to assessing viability and net economic benefits. Viability Viability (also referred to as sustainability) is to be assessed to provide reasonable assurance of the applicant s ability to remain a going concern and the capability of the organization to complete the project as detailed in the ACOA contribution agreement. According to program guidelines, non-commercial viability is to be assessed based on a consideration of the following factors: the demonstrated expertise and capability of the non-commercial operation, with an emphasis placed on enhancing existing strengths; Page 12 Internal Audit Directorate
the contribution of the target group toward costs; and the likelihood that the provision of the service can continue without ongoing government financial assistance. Net Economic Benefit The benefits of a project are to be assessed in terms of the intended results or impact and the contribution of the net economic benefit to the region. Based on non-commercial project guidelines, an assessment of economic benefits should consider the following factors: the level of membership of the association or the size of the target group; the availability of the service from other commercial or non-commercial operations in the region; the relative priority of the service to the commercial viability of the target group; the potential to replace imports of the service into the region or to develop exports; and the ability of the target group to pay commercial prices for the service in question. The audit concluded that WBI and YEDI projects were assessed using viability and net economic benefit criteria detailed in the guidelines, as outlined above. It was, however, less evident that the same assessment criteria were used for BDP projects. In particular, for most BDP non-commercial EBSD projects, specific assessment considerations were not often documented when determining viability and net economic benefits. For example, consideration of the ability of the target group to pay commercial prices for the service in question was evident in very few cases. Recommendations: Expected results should be directly linked to the discussion of viability and net economic benefits on PSFs. Where key consideration factors for both viability and net economic benefits are specifically outlined in program guidelines, they should be referenced in project files to enhance the assessment process. Internal Audit Direcorate Page 13
OBJECTIVE 2 Provide assurance that project approvals and amendments are in accordance with established delegated authorities and that related contracts contain provisions as required by program policies and guidelines. Criterion - Approved delegations of authority are observed for all project approvals, signed Letters of Offer are on file, and project amendments are completed in accordance with program guidelines and authorities. PSFs are used to document the project approval process. Based on the project files examined, it was evident that approvals were made in accordance with delegated signing authorities. Signed Letters of Offer were retained on file for all contributions, and contain Statements of Work, General Conditions, and relevant specific conditions arising from the project approval. Project amendments, where needed, were also completed in accordance with delegated signing authorities. OBJECTIVE 3 Provide assurance that the Agency exercises due diligence in disbursing program funds to recipients. Criterion - Proper information is gathered, reviewed and approved in support of the payments of advances and claimed costs. Similar payment processes are used by ACOA for the disbursement of contributions to both commercial and non-commercial applicants. In most cases, costs are supported by detailed listings of expenditures, compliance with required financial conditions is verified, and applicant-prepared progress reports and certifications are obtained. Normally, an initial review of claims by an Account Assistant is performed to ensure the completeness and accuracy of information provided by applicants, and to organize the information for Account Managers subsequent review and approval. In our view, this represents an efficient mechanism to eliminate clerical tasks from Account Managers workloads and provide for a sufficient segregation of duties, which serves as an effective control. Consistent with the results of two previous audits (BDP and Innovation), we noted that in cases where the Account Managers did not sign the payment checklists, it was difficult to determine the extent to which the Account Managers reviewed claimed costs, or the extent to which claimed costs were analyzed in relation to eligible cost categories as detailed in Statements of Work. Page 14 Internal Audit Directorate
Consulting Advisory Service (CAS) Projects Grants and Contributions (G&C) Contributions are used to support the Women in Business CAS projects in Prince Edward Island and New Brunswick. 1 In these regions, external associations receive project funding through contribution agreements with which they offer CAS support to women business owners who are eligible for the program s various components. Projects are assessed and administered by the association; however, ACOA maintains a level of accountability through monitoring of claims, eligibility of recipient clients, project costs and final products. The audit concluded that claims are generally not supported by sufficient information to allow for a detailed review of the eligibility of costs claimed or the eligibility of the recipient client. In this regard, it was difficult to determine the extent to which ACOA was able to review and draw a conclusion on the acceptability of the claims submitted for payment. Recommendations: The Account Managers review of claimed costs with respect to amounts, composition and eligibility should be documented, and managers should sign the payment checklist indicating their review and approval of work performed by Account Assistants. (Note: This recommendation is consistent with two prior audits and is being addressed in the BDP Management Action Plan.) For Women in Business CAS projects, claims should be submitted with sufficient information to allow Account Managers to perform a detailed review of eligibility and compliance with contract conditions, including invoice numbers, names of consultants and recipient clients, project description, cost amounts with applicable taxes and project rate of assistance. OBJECTIVE 4 Provide assurance that reporting and monitoring activities are adequate to meet program requirements established for each project during the assessment and approval process. Criterion 1 - Financial statements are obtained from applicants on an annual basis (in accordance with contract requirements). For those projects where the submission of externally prepared financial statements was a contractual condition, we found that the financial statements 1 Women in Business CAS project ceased in September 2006. Internal Audit Direcorate Page 15
had been received and the summarized financial information had been entered into QAccess in the majority of cases. Criterion 2 - Client reporting is completed in accordance with program and contract requirements. The monitoring of non-commercial projects is dependent on the payment process, while formal monitoring reports are required for those projects involving repayable contributions. Applicant-prepared progress reports are normally required by the Agency to substantiate the payment of interim claims, and final project reports are required to substantiate final disbursements. For the project files examined as part of this audit, we found that, in the majority of cases, progress and final reports were submitted in accordance with the reporting requirements established in the Letters of Offer. Criterion 3 - Final reports are obtained and evaluated at the conclusion of each project. Due to the soft-cost nature of EBSD projects, the results are often intangible and can be difficult to measure. Reporting on the results is an Agency requirement, however, and must be performed in accordance with initiative objectives. Upon receipt of the client s final report, the achievements/benefits of the project are evaluated by the Account Manager. Annually, EBSD project achievements are submitted for Agency reporting in the Departmental Performance Report (DPR). In the majority of cases, final reports were provided by the clients, but it was often not often clear that the reports were evaluated by the Account Managers. We were unable to find project achievements/benefits recorded in QAccess or documented on the project files; however, there was evidence of a manual system for capturing data in each region. At the time of the audit, it was apparent that limitations inherent in the Agency s corporate system hinder the recording of EBSD project results in QAccess. BEST PRACTICE A final report completion certificate is used at ACOA NL to indicate receipt, review and acceptance of final reports by Account Managers. We believe that the addition of a section identifying project benefits would further enhance this practice. BEST PRACTICE WBI and YEDI Letters of Offer include specific reporting requirements for progress and final reports. This practice provides clarity for both the client and the Agency, as illustrated bellow. Page 16 Internal Audit Directorate
The report shall include: (a) a statistical account of the number of clients who have availed of the Applicant's services resulting from the Project. (b) a report of all seminars and workshops facilitated by the Applicant as a result of the Project. (c) a report on the promotional activities undertaken in reference to the Project. (d) a summary of networking activities relevant to the Project including any newsletters, special events, etc. (e) all other information relevant to providing a comprehensive report of the reporting period's activities. CAS Projects Operating and Maintenance (O&M) In the regions of Newfoundland and Labrador, Nova Scotia and at ECBC, CAS projects are supported through ACOA s O&M funding. The CAS projects in these regions are assessed and administered by Agency Account Managers in accordance with WBI CAS guidelines. The audit concluded that project files were well documented and included the consultants final reports. It was less evident, however, that the Agency documented follow-up undertaken in respect of client action arising from CAS projects, client satisfaction with the services received, or benefits realized. BEST PRACTICE In Nova Scotia, a detailed WBI CAS checklist is used to ensure pre-determined processes have been followed and all relevant documentation is on file. The checklist is attached to the inside cover of each project file folder for ease of reference. CAS Projects Grants and Contributions (G&C) In some regions, associations receive project funding that can be used to offer CAS to women business owners. ACOA s responsibilities remain the same as for all other contributions, including defining expected results, establishing the means for measuring results and monitoring project achievements. With respect to the monitoring of Women in Business CAS projects, we found limited evidence that the Agency evaluated progress or final reports, concluded on project achievements, or reviewed client evaluations of the CAS services or products received. Recommendations: Account Managers should document their evaluations and acceptance of reports submitted pursuant to EBSD requirements, including specific conclusions on results achieved. Internal Audit Direcorate Page 17
The functionality of the Agency s corporate system should be reviewed with the intention of facilitating the collection of EBSD project achievements. Project achievements or benefits should be recorded in QAccess or the project files. For CAS projects, Account Managers should perform a periodic review of final products provided by consultants and the recipient client s satisfaction with the product/service. Page 18 Internal Audit Directorate