Legislative Audit Division State of Montana

Transcription

1 Legislative Audit Division State of Montana January 2006 Report to the Legislature Performance Audit Small Business Development Centers Business Resources Division Department of Commerce This report provides findings and recommendations developed as a result of a performance audit of the Small Business Development Centers program. Findings and recommendations address: Provision of SBDC services and alternative service providers. Compliance with federal regulations. Trends in program counseling and training activities. Program services for veterans and American Indians. Economic trends and program service areas. Program outputs and outcomes reporting. Direct comments/inquiries to: Legislative Audit Division Room 160, State Capitol PO Box P-04 Helena MT Help eliminate fraud, waste, and abuse in state government. Call the Fraud Hotline at statewide or in Helena.

2 PERFORMANCE AUDITS Performance audits conducted by the Legislative Audit Division are designed to assess state government operations. From the audit work, a determination is made as to whether agencies and programs are accomplishing their purposes, and whether they can do so with greater efficiency and economy. The audit work is conducted in accordance with audit standards set forth by the United States Government Accountability Office. Members of the performance audit staff hold degrees in disciplines appropriate to the audit process. Areas of expertise include business and public administration, mathematics, statistics, economics, political science, criminal justice, computer science, education, and biology. Performance audits are performed at the request of the Legislative Audit Committee, which is a bicameral and bipartisan standing committee of the Montana Legislature. The committee consists of six members of the Senate and six members of the House of Representatives. MEMBERS OF THE LEGISLATIVE AUDIT COMMITTEE Senator Joe Balyeat, Vice Chair Senator John Brueggeman Senator Jim Elliott Senator Dan Harrington Senator Lynda Moss Senator Corey Stapleton Representative Dee Brown Representative Hal Jacobson Representative Christine Kaufmann Representative Scott Mendenhall Representative John Musgrove, Chair Representative Janna Taylor

3 LEGISLATIVE AUDIT DIVISION Scott A. Seacat, Legislative Auditor John W. Northey, Legal Counsel Deputy Legislative Auditors: Jim Pellegrini, Performance Audit Tori Hunthausen, IS Audit & Operations James Gillett, Financial-Compliance Audit January 2006 The Legislative Audit Committee of the Montana State Legislature: This is our performance audit of Small Business Development Centers (SBDC), a program administered by the Department of Commerce. This report includes information on continuing public provision of SBDC services, availability of alternative service providers, program administration and compliance with federal regulations, program activity and service provision trends, analysis of economic trends in program service areas, and program outcomes reporting. A written response from the Department of Commerce is included at the end of the report. We wish to express our appreciation to SBDC counselors and Department of Commerce staff for their cooperation and assistance during the audit. Respectfully submitted, /s/ Scott A. Seacat Scott A. Seacat Legislative Auditor Room 160, State Capitol Building PO Box Helena, MT Phone (406) FAX (406) [email protected]

4 Legislative Audit Division Performance Audit Small Business Development Centers Business Resources Division Department of Commerce Members of the audit staff involved in this audit were Lisa Blanford and Angus K. Maciver.

5 Table of Contents List of Figures and Tables... iv Appointed and Administrative Officials... v Glossary of Summary... vi Report Summary...S-1 Chapter I - Introduction & Background... 1 Introduction... 1 Audit Objectives... 1 Audit Approach... 2 Background Information... 2 Statutory References... 3 SBDC Network Structure... 4 SBDC Advisory Board... 5 SBDC Counseling and Training Activities... 6 Federal SBDC Administration... 6 Program Funding... 7 Report Organization... 9 Chapter II - Evaluation of SBDC Service Provision Introduction Comparison Group Study of SBDC Economic Impact Comparison Group Design Trends in Reported Taxable Income Trends in Reported Employment Data Trends for Population Sub-Sectors SBDC Business Benefit from Enhanced Productivity SBDC Program Meets Its Primary Objective SBDC Service Provision Program Provides Basic Business Planning Assistance Private Sector Service Providers Calculating the Cost of Private Services Private Sector Services Are Available, But Costly Other Service Providers Locations of Public Sector Providers Overlap Exists Among Service Providers Overall Conclusion SBDC Program Could Improve Operational Efficiency and Effectiveness Chapter III - Program Compliance & Administration Introduction Federal Administrative & Compliance Standards Program Meets Federal Compliance Standards Program Oversight Procedures Contractual Relationship Between Lead and Sub-Centers Lead Center Meets Federal Compliance and Oversight Standards Page i

6 Table of Contents SBA Business Size Regulations Trends in Client Business Types Some Businesses Exceed Size Regulations SBDC Over-Reported Some Client Data Program Managers Need to Improve Oversight of Client Records Comparison with Other States University System Administration of SBDC Programs Network Structures in Western States Federal and State Funding of SBDC Overall Conclusion Chapter IV Counseling & Training Activities Introduction Data Analysis Definitions and Parameters Sub-Center Documentation Procedures Counseling & Training Activities are Documented Correctly Counseling and Training Activity Trends Number of Clients Counseled is Decreasing Training Events and Attendance are Increasing Program Uses Training to Screen Participants Analysis of Training Event Locations Program Should Address Availability of Training in Some Locations Client Demographic Trends Special Emphasis Groups Trends in Veteran Participation Program Should Conduct Further Analysis of Veteran Participation American Indian Participation Reservation Training Activities Indian Economic Development Initiatives SBDC Program Should Address Reservation Activities Overall Conclusion Chapter V - Economic Trends & SBDC Service Areas Introduction SBDC Client Locations SBDC Serves a Predominately Urban Clientele Small Business Activity in SBDC Service Areas Analysis of County Economic Growth Trends Trends in SBDC Client Activities Observed Trends in Sub-Center Performance Several Factors Could Affect Sub-Center Workload Program Should Address Resource Allocation Effects of Some Programmatic Trends are Uncertain Further Monitoring and Resource Allocation Changes are Necessary Page ii

7 Table of Contents Client Activity Trends Not Commensurate With Prevailing Economic Conditions Intervention Required to Correct Resource Allocation Overall Conclusion Chapter VI - Program Outcomes Reporting Introduction Program Outputs and Performance Goals Analysis of SBDC Outputs Performance Targets Adjustments in Performance Targets are Necessary Verifying SBDC Outcomes and Economic Impact Program Approach to Outcomes Reporting is Conservative Outcomes Data Cannot be Independently Verified Existing Economic Impact Studies Rely on Self-Reported Data SBDC Program Outcomes Reporting Reported Outcomes Data and Trends Reported Outcomes in Other States Improvements in Performance Measurement System Needed Overall Conclusion Appendix A - Audit Approach...A-1 Audit Scope...A-1 Audit Methodologies...A-1 Audit Planning...A-1 SBDC Service Provision...A-2 Alternative Service Providers...A-2 Administrative Compliance and Oversight...A-3 SBDC Counseling and Training Activities...A-4 Economic Trends in Program Serviec Areas...A-4 Program Outputs and Outcomes Reporting...A-5 Department Response...B-1 Department of Commerce...B-3 Page iii

8 List of Figures and Tables FIGURES Figure 1 Montana SBDC Regional Network... 4 Figure 2 Report Organization Figure 3 Median Taxable Business Income for Comparison Groups Figure 4 Median Annual Payroll per Employee for Comparison Groups Figure 5 46 Montana Communities With Business Technical Assistance Resources Figure 6 SBDC Clients Counseled and Average Hours per Client Figure 7 Density Distribution of SBDC Clients Figure 8 Growth in Business Establishments by SBDC Service Area Figure 9 Median Annual Variance in Sub-Center Client Activity Performance Level Figure 10 Trends for Selected Output Performance Measures Figure 11 Trends in Reported Outcomes Data Figure 12 Selected Outcomes Data for Other Sources TABLES Table 1 Basic Management Information for SBDC... 3 Table 2 SBDC Funding Sources and Amounts... 8 Table 3 Lead Center and Sub-Center Budgets... 9 Table 4 Type of Counseling Services Provided Table 5 SBDC Network Structures in Montana and Other Western States Table 6 State and Non-Governmental Funding of SBDC Programs Table 7 Veteran Participation By SBDC Sub-Center Table 8 SBDC Training Events on Indian Reservations Page iv

9 Appointed and Administrative Officials Department of Commerce Anthony J. Priete, Director Andy Poole, Administrator, Business Resources Division Ann Desch, State Director, Small Business Development Centers Page v

10 Glossary of Terms Montana Department of Commerce Business Resources Division SBDC...Small Business Development Centers CRDC...Certified Regional Development Corporations RDO...Regional Development Officers Federal Government SBA...Small Business Administration USDA...United States Department of Agriculture RC&D...Resource Conservation and Development Districts PTAC...Procurement Technical Assistance Centers TBIC...Tribal Business Information Centers Other MMEC...Montana Manufacturing Extension Center SCORE...Service Corps of Retired Executives ASBDC...Association of Small Business Development Centers Page vi

11 Report Summary Introduction The Small Business Development Center (SBDC) program provides business technical assistance through individual counseling and group training. Services are provided through a network of 10 regional sub-centers and the program is administered through a lead center based in the Montana Department of Commerce (the department). SBDC is a nationwide program and the Small Business Administration (SBA) is responsible for program administration at the federal level. SBDC Program Meets Overall Objectives Audit findings and conclusions show the SBDC program is meeting its overall goals and provides benefits to businesses. The program is providing benefits to businesses receiving counseling services. We also conclude that private sector alternatives to program services are either unavailable or unaffordable. We did identify areas where the efficiency or effectiveness of program operations could be improved. Specifically, we address improvements in compliance with federal regulations, delivery of services to special emphasis groups, network resource allocation, and performance measurement procedures. Background Information Montana SBDC Network State and Federal SBDC Administration SBDC was developed to provide small businesses professional counseling and training services they could not otherwise access or afford. Services are delivered through individual sessions with a SBDC counselor, or through group training workshops organized by the regional sub-centers. SBDC sub-centers are located in Kalispell, Missoula, Butte, Helena, Bozeman, Great Falls, Havre, Billings, Colstrip and Wolf Point. Sub-centers are located in host agencies contracting with the department to deliver SBDC services within the region. Nine of the host agencies are local economic development corporations and one (Kalispell) is a chamber of commerce organization. The SBDC lead center is located in the department s Business Resources Division. The lead center employs 2.5 FTE to administer the program, including the State Director responsible for general management and oversight of the whole network. Federal oversight is exercised through SBA state and regional offices. The Association of Small Business Development Page S-1

12 Report Summary Centers (ASBDC) represents local lead centers and networks in negotiations with the SBA and provides members with information, advice and other services. Program Funding SBDC is funded through federal, state and local (host agency) contributions. SBA makes an annual grant to the lead center. The annual grant amount is currently $500,000. The total SBDC budget for 2005 was approximately $1.4 million. The majority of the budgeted funds for the SBDC program in Montana are contributed by host agencies (47 percent), followed by the federal and state government contributions (35 and 18 percent respectively.) SBDC Service Provision Our first audit objective was to determine if the SBDC program provides services that are beneficial to participating businesses and do not duplicate services that are already available. Comparison Group Study of SBDC Economic Impact We conducted a comparison group study to estimate what effects participating in the SBDC program has on client businesses. The comparison group study looked at changes in income and employment for two groups. The first group received SBDC counseling services and the second group received no assistance from the program. Using data from the Department of Revenue and the Department of Labor and Industry, we tracked and compared growth in taxable business income and employment data for both groups. Overall, the comparison group study shows the program is delivering benefits to participants. In terms of growth in annual taxable income and annual payroll per employee, participation in the SBDC program appears to provide small businesses in Montana with significant benefits. Alternative Service Providers Small businesses seeking technical assistance have access to other service providers in both the public and private sectors. The benefits provided by SBDC services are important, but the program must also be able to demonstrate that alternative programs or organizations Page S-2

13 Report Summary cannot provide similar services and benefits or can do so at reduced cost. Private Sector Service Providers Based on reported hourly billable rate of $150, private sector management consultant services would cost $672 for the average SBDC client. In addition, the services typically being sought by SBDC businesses are related to basic business planning functions. Private sector management consultants typically concentrate on more specialized services. Given the types of services generally offered by management consultants and the costs involved in retaining the services of specialized firms, it is unlikely SBDC functions would be provided solely through the private sector. Other Service Providers Business technical assistance is also provided by public sector organizations, public-private partnerships, and non-profit groups. Review of information from other providers showed each program or service has a specific mission, but in some cases there is not always a clear boundary established between SBDC resources and services and those of other providers. However, despite overlaps in service provision, the SBDC program occupies a niche position and its services are not entirely replicated by any other provider. Program Compliance and Administration Audit objectives addressed the extent to which the SBDC program meets federal compliance standards and the degree of consistency between Montana s administrative structure and programs in other states. Some Businesses Exceed Size Regulations As with all SBA-administered programs, SBDC operations are governed by federal regulations relating to the size of the businesses receiving assistance. To determine whether businesses participating in the Montana SBDC program meet size regulations, we reviewed program records. Eight of the fifteen businesses included in our review did not meet SBA business size regulations. Improving awareness of the SBA business size regulations among sub-center directors and lead center oversight of sub-center data reporting could prevent future occurrences of non-complying businesses accessing SBDC services. Page S-3

14 Report Summary Comparison With Other States All 50 states currently participate in the SBDC program and there are over 1,100 sub-centers in locations around the country. Montana s SBDC network structure is consistent with the state s low population density and high land area as compared with other western states. Federal law establishes the distribution of funding to state SBDC programs and also sets minimum matching requirements for recipient organizations. When compared with other states, Montana has been effective in using host agency contributions to leverage state funding and increase resources available for SBDC program operations Counseling and Training Activities SBDC counselors provide two primary services to clients; individual counseling sessions and group training events. Audit objectives addressed how well the program serves the state s small business community. Trends in Counseling and Training Activities Analysis of trend data for clients counseled between 1998 and 2004 showed that there has been a gradual decline in the number of clients counseled by the program. As the number of clients counseled has decreased, the average number of counseling hours per client has increased. We also analyzed training data for 1998 through 2004 to identify how many events were held by each sub-center in each year. There are clear trends indicating the number of events is increasing across the network. Discussions with SBDC counselors and lead center staff indicate the program is using training events to screen for participants with more advanced business plans. As client workloads decline, counselors can spend more time counseling individual clients. The Montana SBDC program is pursuing a strategy of reducing client workloads through time-intensive counseling and expanded training opportunities. Training Event Locations Additional analysis of training addressed locations of events in cities and towns around the state. In general, sub-centers ensured training events were held in different communities around the service area. The exception to this trend was the Billings sub-center. The department should ensure cooperative agreements and host agency Page S-4

15 Report Summary contracts address provision of training opportunities in outlying communities within SBDC service areas. Program Should Conduct Further Analysis of Veteran Participation Outreach to veterans has been identified by SBA and the Montana SBDC program as a special emphasis initiative. Veterans make-up around 11 percent of the state s working-age population and around percent of program participants. However, some disparities were observed at the sub-center level through analysis of data for veteran populations in specific counties. Program managers should continue to monitor the effectiveness of out-reach efforts targeting the state s military veterans. In particular, the effectiveness of outreach efforts in larger cities should be the subject of more detailed analysis to determine if the veteran population in these areas is being effectively served by the program. SBDC Program Should Address Reservation Activities The other special emphasis initiative group targeted by Montana s SBDC program is the American Indian population. Between 1998 and 2004, SBDC participation has declined by around 3 percent annually, but for residents of the state s Indian reservations, the decline is closer to 6 percent. Review of training records for 1998 to 2004 showed that efforts on some of the state s reservations have been inconsistent. By ensuring an enhanced training schedule is properly implemented for Indian reservations, the SBDC sub-centers may be able to raise awareness of the services and resources available through the program. SBDC program services could, potentially, provide a valuable resource for tribal governments addressing the provision of business technical assistance services. Addressing how SBDC resources can help ongoing efforts to promote Indian economic development could clarify the program s role on reservations and relations with tribal and other technical assistance providers. Economic Trends and SBDC Service Areas The SBDC network is configured to deliver services to a specific population; the state s small business community. Our audit objectives included assessment of how closely the structure and organization of the SBDC network corresponds with the dynamics of the private sector economy. Page S-5

16 Report Summary Analysis of County Economic Growth Trends Economic growth in the state has not been even in recent years and counties experiencing significant growth rates are concentrated in the south and west of Montana. SBDC service areas have experienced varying growth rates in the number of business establishments. Three sub-centers (Missoula, Kalispell and Billings) have performed below the ideal level of client activity based on economic trends within their service areas. These sub-centers have served fewer clients than would otherwise be expected based on the projected levels of activity resulting from economic trends in their service areas. Program Should Address Resource Allocation As change in the wider economy accelerates, resource allocation within the SBDC network will become increasingly difficult to manage. As measured against a projected client base, SBDC subcenters in fast-growth urban areas are failing to maintain levels of activity sufficient to meet demand for services. To avoid difficulties in serving clients in the state s urban areas, the department needs to adjust funding distribution and allocate resources to SBDC subcenters based on prevailing economic trends within service areas. The delivering of program services can be improved by allocating funding and other resources based on demand. Program Outcomes Reporting The program uses a performance measurement system to track both outputs and outcomes. Audit objectives included assessment of output targets in the program s performance measurement system and the accuracy and reliability of program outcomes data. Program Outputs and Performance Goals As part of annual cooperative agreements with the SBA, targets are established for activity levels within each sub-center. The program s use of performance targets has been improving, but some disparities still exist between growth trends for target and actual performance data. Performance targets should be monitored to ensure they continue to reflect changes in program conditions. Adjustments should be made in performance targets to ensure they are both realistically achievable and sufficiently challenging. Page S-6

17 Report Summary Verifying SBDC Outcomes and Economic Impact We reviewed program procedures for verifying reported client outcomes and estimating economic impacts through surveys and other studies. Although the program takes a conservative approach to reporting outcomes, significant improvements are needed in verifying the validity of this data. Essentially, the program relies on good faith estimates provided by clients to determine the outcomes attributable to SBDC services. Improvements in Performance Measurement System are needed Program managers should track and report client outcomes, but the success of these efforts relies on consistency in the methodologies used to define and report outcomes. Improvements in data definition and reporting parameters have already been started through the SBA and the ASBDC. Continuing implementation of standardized outcomes data definitions and reporting parameters should improve the quality and reliability of reported outcomes data. Page S-7

18 Chapter I - Introduction & Background Introduction The Small Business Development Center (SBDC) program provides business technical assistance through individual counseling and group training for existing business owners or people interested in starting a business. Services are provided through a network of 10 regional sub-centers around the state and the program is administered through a lead center based in the Montana Department of Commerce (the department). SBDC is a nationwide program and is jointly funded by the federal government, and state and local partners. The Small Business Administration (SBA) is responsible for program administration at the federal level. Audit Objectives The Legislative Audit Committee prioritized a performance audit of the SBDC program for the 2005 biennium. In response to issues identified by the Legislative Audit Committee, we developed the following five main audit objectives: 1) Determine if the SBDC program benefits participating businesses and does not duplicate services already available in the public and private sectors. 2) Determine if the Montana SBDC lead center performs its duties in accordance with applicable federal laws and regulations and the program s organizational structure is consistent with networks in other states. 3) Determine what types and levels of services are provided to small businesses through the SBDC program and if the program effectively serves the state s small business community. 4) Determine if levels of activity for SBDC sub-centers are commensurate with prevailing economic trends in their respective service areas. 5) Determine if the program s use of outputs and outcomes data provide an accurate and reliable basis for measuring performance. The findings and conclusions presented in chapter II show the SBDC program is meeting its overall goals and provides benefits to participating businesses. Methodologies conducted under our first audit objective included a comparison group study of businesses participating in the SBDC program. We conclude the program is providing benefits to businesses receiving counseling services. We Page 1

19 Chapter I - Introduction & Background also conclude that alternatives to program services are either unavailable or unaffordable for many small businesses. In subsequent chapters, we identify areas where the efficiency or effectiveness of program operations could be improved. Specifically, we address improvements in compliance with federal regulations, delivery of services to special emphasis groups, network resource allocation, and performance measurement procedures. Background and funding information is provided in the sections below. Audit Approach Unless otherwise stated, analysis of program data addresses calendar years 1998 through This timeframe was established to ensure sufficient historical data was available to identify trends in program activities. Further information relating to audit scope, scope exclusions, and methodologies relating to audit objectives and subobjectives are contained in Appendix A. Background Information SBDC is a business technical assistance program. Business technical assistance consists of non-financial services designed to help businesses overcome problems and promote growth. The program was developed to provide small businesses professional counseling and training services they could not otherwise access or afford. SBDC assistance generally includes the formation and development of business plans, financial projections, and marketing strategies for small businesses. Services are delivered through individual sessions with a SBDC counselor, or through group training workshops organized by the regional sub-centers. The SBDC program also acts as an information resource and referral for other specialist services and programs aimed at small businesses. The SBDC program originally developed from technical assistance programs in land grant universities. Small business technical assistance was seen as a natural extension of the mission of the land grant universities. Montana s SBDC program is significantly different from the majority of other states in that sub-centers are not located in universities or other institutions of higher education. Montana and several other states have been granted exceptions under Page 2

20 Chapter I - Introduction & Background federal law allowing them to locate SBDC lead centers in nonuniversity organizations or agencies. The following table summarizes basic management information for the SBDC. Table 1 Basic Management Information for SBDC Calendar Year 2004 Annual Budget $1.4 Million SBDC Counselors 13 Department Administrative Staff 2.5 FTE Clients Counseled 1,406 Counseling Sessions 2,712 Counseling Hours 5,576 Training Events 240 Training Event Attendance 2,534 Source: Compiled by the Legislative Audit Division from department records. Statutory References The SBDC program is established in the Small Business Act (Public Law ), which authorizes the SBA to make grants to states for administration of the program. The Small Business Act also outlines SBDC grant funding formulas, state matching requirements, required activities and services, coordination with other federal small business programs, and oversight procedures. The Code of Federal Regulations (Title 13, Chapter 1, Part 130), outlines specific operational requirements for the states administering the program. Federal regulations address the role of the lead center and subcenters, operating requirements for sub-centers, SBDC service provision, responsibilities of lead center directors, establishment of program advisory boards, grant application and matching funds requirements, and program audit and oversight procedures. Page 3

21 Chapter I - Introduction & Background The SBDC program has no operating or other requirements established in Montana statute or administrative rule. Section (8) (b), MCA, statutorily appropriates funding from interest income of the coal severance tax permanent fund to the SBDC program. SBDC Network Structure Montana s SBDC sub-centers serve 10 regions around the state established by the department. Service areas are defined by county boundaries as shown in the following figure. Figure 1 Montana SBDC Regional Network Source: Compiled by the Legislative Audit Division from department records. SBDC sub-centers are located in Kalispell, Missoula, Butte, Helena, Bozeman, Great Falls, Havre, Billings, Colstrip and Wolf Point. Sub-centers are located in host agencies contracting with the department to deliver SBDC services within the region. Nine of the host agencies are local economic development corporations and one (Kalispell) is a chamber of commerce organization. All host agencies are required to assign one full-time employee as the SBDC Director. The sub-center director is the primary SBDC counselor for Page 4

22 Chapter I - Introduction & Background the region, organizes training events, and is responsible for subcenter administration. For program year 2005, three sub-centers were budgeted for an additional SBDC counselor, Billings, Missoula and Bozeman. The SBDC lead center is located in the department s Business Resources Division. The lead center employs 2.5 FTE to administer the program, including the State Director responsible for general management and oversight of the whole network. Montana s State Director is also responsible for management oversight of three other department programs providing services or resources to small businesses (Small Business Innovation Research, NX Level Training, and Main Street programs). The lead center is responsible for relations with the federal government, managing the contracts between the department and host agencies, distributing grant monies, developing strategic plans and other program management tools, compiling and reporting program statistics and accomplishments, and providing general oversight of activities within SBDC subcenters. The SBDC lead center is also responsible for maintaining relations with the Association of Small Business Development Centers (ASBDC). The ASBDC is a national organization representing all SBDC participants (sub-centers and Host agencies) around the country. ASBDC administers a certification program to ensure SBDC state networks meet certain standards and represents state networks in negotiations with the SBA SBDC Advisory Board Lead centers are required by SBA to establish an advisory board to provide guidance for the SBDC program. Advisory boards must include small business owners and representatives from the network s entire area of service. Montana s SBDC program has established an advisory board consisting of 14 members, the majority of whom are small business owners. The advisory board meets once per year to review program operations and provide input and direction for program managers. Page 5

23 Chapter I - Introduction & Background SBDC Counseling and Training Activities The SBDC program refers to prospective business operators as preventure or start-up clients i.e. individuals with a desire to form a business or a commercial idea (this group generally includes small businesses in the first six months of existence). The program differentiates between pre-venture and existing businesses because the types of services required tend to be different for these two groups. Pre-venture clients often require basic information on issues such as registering a business entity, preparing a business plan, understanding the regulatory requirements for new businesses, analyzing local markets and competitors, and securing business startup loans. As a business develops, SBDC counselors offer more specialized advice on issues such as access to development capital/financing, development of marketing strategies, personnel management, and regulatory implications for specific industry sectors. Each SBDC counselor is available for both individual counseling and group training. Sub-centers can also use volunteers from the local community to teach classes. Examples include employees of local banks or credit unions teaching classes on small business loans/financing, local business owners with experience in specific subject areas, or members of SCORE (Service Corps of Retired Executives) local chapters. Most of the counseling and training opportunities offered by SBDCs are free to participants. For some activities, sub-centers require a fee, although the fee amount is usually minimal. Federal SBDC Administration Federal oversight is exercised through SBA state and regional offices. Montana s state office is located in Helena and reports through the SBA regional office in Denver, Colorado. SBA enters into annual cooperative agreements with states administering SBDC networks. Cooperative agreements cover a calendar year and are reviewed and revised annually. Because the national program was originally developed as a partnership between an existing network of local centers and a federal agency, the various institutions and organizations responsible for local service delivery formed a national association to represent their interests. ASBDC represents local lead centers and networks in negotiations with the SBA and provides Page 6

24 Chapter I - Introduction & Background members with information, advice and other services. ASBDC accreditation is necessary before SBA funding is made available to states. Program Funding SBDC is funded through federal, state and local (host agency) contributions. The funding process begins with an annual grant from the SBA made to the lead center (through the department). The annual grant amount is currently $500,000. Montana is categorized by SBA as a small state and receives the minimum funding amount. The state contributes funding through the General Fund and a statutory appropriation from coal severance tax permanent trust fund interest income. Funds distributed to local sub-centers need to be matched 100 percent by host agencies. At least 50 percent of the host agency s match needs to be cash; the remaining 50 percent can be allowable cash, in-kind contributions, or indirect costs. The following table shows the funding amounts and sources for the SBDC program in Table 2 SBDC Program Funding Sources and Amounts Calendar Year 2005 Funding Source Amount Fund Source Percentage Federal Grant $500, % State General Fund $122,000 State Coal Trust Interest Income $125,000 Local Cash Match $506,135 Local In-Kind Contributions $159, % 47 % Total Funding $1,412,956 Source: Compiled by the Legislative Audit Division from department records. The total SBDC budget for 2005 was approximately $1.4 million. As shown, the majority of the budgeted funds for the SBDC program Page 7

25 Chapter I - Introduction & Background in Montana are contributed by host agencies (47 percent), followed by the federal and state government contributions (35 and 18 percent respectively.) The federal and state-sourced components of the total budget are distributed to both the lead center in Helena and the local SBDC sub-centers. The lead center retains $100,000 of the SBA grant to cover administrative costs. The remaining $400,000 is distributed to subcenters to fund activities. Sub-centers receive equal shares of SBA funding ($40,000 each), and therefore must provide a minimum cash match of $20,000 and a further $20,000 in other allowed contributions. Additional funding for sub-centers is provided through the statutory appropriation of $125,000. The Helena lead center and nine sub-centers receive an additional $10,000 in funding from the statutory appropriation and one sub-center (Billings) receives $20,000 (the Billings sub-center receives additional funding to cover costs associated with increased workload.) Additional lead center funding comes through the department s $122,000 general fund appropriation for the Business Resources Division. The next table shows budgets for the lead center and sub-centers for program year Page 8

26 Chapter I - Introduction & Background Table 3 Lead Center and Sub-Center Budgets Calendar Year 2005 Network Sub-Center Budgeted Amount Percentage of Total Billings $135, % Bozeman $143, % Butte $107, % Colstrip $100, % Great Falls $114, % Havre $106, % Helena $108, % Kalispell $120, % Missoula $156, % Wolf Point $100, % Lead Center $220, % Total $1,412, % Source: Compiled by the Legislative Audit Division from department records. Federal and state funding is generally distributed on an equal basis between the 10 sub-centers. Differences in budget amounts for the sub-centers reflect local contributions for increased staff numbers and scope of activities, or funding available from other sources. Report Organization The following figure illustrates organization of remaining chapters for this report. Page 9

27 Chapter I - Introduction & Background Figure 2 Report Organization Chapter II Chapter III Chapter IV Chapter V Chapter VI Program Effectiveness Program Compliance Counseling and Training Activities Economic Trends and Network Structure Outcomes Reporting SBDC Services Benefit Businesses Program Meets Compliance Standards Client Workloads Declining SBDC Clients Mainly Live in Urban Areas Performance Targets Not Realistic Private Sector Provision More Costly Some Businesses Exceed SBA Size Regulations Training Provision Increasing More Economic Growth Occurs in Urban Centers Adjust Performance Targets Multiple Public Sector Providers Exist Improve Monitoring of Client Records Training Events Needed in Some Locations Service Areas not Reflective of Economic Trends Outcomes Reporting Conservative Public Provision of SBDC Services Justified Montana SBDC Administration Consistent With Other States Services for Veteran & American Indian Populations Inconsistent Monitor Trends/Adjust Resource Allocation Methods Outcomes Measures Not Verifiable Analyze Veteran Participation Standardize Outcomes Measures Objectives Conclusions Recommendations Improve Service on Indian Reservations Source: Compiled by the Legislative Audit Division. Page 10

28 Chapter II - Evaluation of SBDC Service Provision Introduction Our first audit objective was to determine if the SBDC program provides services that are beneficial to participating businesses and that these services are otherwise unavailable. Economic development programs need to deliver tangible benefits to participants to be considered successful. For publicly funded economic development efforts aimed at small businesses, the SBA emphasizes it is also important services are either unavailable or unaffordable from alternative public and private sector providers. Therefore, our objectives also included assessment of the extent to which SBDC services compete with private sector providers or duplicate public sector or non-profit programs. The following sections discuss audit findings in these areas. Comparison Group Study of SBDC Economic Impact We can determine if the SBDC program is successful by looking at changes in income and employment for businesses that used program services. The best way of using this business data is to conduct a study similar to a scientific experiment, where changes are observed under controlled conditions. We conducted a comparison group study to estimate what effects participating in the SBDC program has on client businesses. The comparison group study looked at changes in income and employment for two groups. The first group received SBDC counseling services and the second group received no assistance from the program. The businesses which did not receive assistance from SBDC were not selected randomly. We chose businesses which were very similar to the companies participating in the program. By comparing two very similar groups of businesses, we can be more confident that changes were the result of SBDC services, rather than some other cause. Comparison Group Design For our study, we used businesses that received at least three hours of counseling from the SBDC program between 1998 and 2000 (referred to as the program group). We then matched each of the SBDC businesses with companies that have never received any counseling from the program (referred to as the comparison group.) The matching process involved finding a company operating a Page 11

29 Chapter II Evaluation of SBDC Service Provision similar type of business, in a similar location, and for a similar number of years. The following example shows how we matched two similar businesses to ensure a like-to-like comparison: The program group contains an auto repair business, which received five hours of SBDC counseling in August The business is located in Conrad, Montana, and has been in operation since 1995 (approximately ten years.) The comparison group, therefore, contains an auto repair businesses, which has received no SBDC counseling services. The matching business is located in Red Lodge, Montana (control group businesses do not have to be located in the same town, but should be from a community of comparable size), and has been in operation for a similar number of years. Using data from the Department of Revenue and the Department of Labor and Industry, we tracked and compared growth in taxable business income and employment data for both groups between 1999 and Comparing growth trends for both groups allows us to determine if SBDC services resulted in benefits for participating businesses. For example, if the auto repair business from the program group grew by 20 percent every year, but the auto repair business in the comparison group grew by only 10 percent, we can assume that some factor caused one business to grow faster than the other. In our study, both businesses are very similar, except for the fact that one received SBDC counseling and the other did not. Therefore, the cause of the difference between growth rates for these two businesses is reasonably likely to be the SBDC services. We can estimate the effects of SBDC program services by looking at the overall growth rates for all the businesses included in our two groups. We obtained data for around 45 SBDC businesses, representing approximately 3 percent of the program s counseling workload between 1999 and Identical data was obtained for 45 comparison group businesses. Trends in Reported Taxable Income The two groups were first analyzed to determine if differences existed in growth rates for taxable business income reported to the Department of Revenue. Only those businesses reporting data for Page 12

30 Chapter II Evaluation of SBDC Service Provision each year between 1999 and 2003 were included. The median taxable business income was calculated for program and comparison groups in each tax year and the trend is illustrated in the following figure. Figure 3 Median Taxable Business Income for Comparison Groups Calendar Years $20,000 Median Taxable Income $15,000 $10,000 $5,000 $ SBDC Businesses Non-SBDC Businesses Source: Compiled by the Legislative Audit Division from Department of Commerce and Department of Revenue records, and from other sources. As shown in the figure, median taxable income for both groups falls through 2001 as the economy enters recession. From 2001 to 2002, median taxable income for both groups increases, but businesses that received SBDC counseling in prior years recover faster and sustain higher levels of taxable income through 2003, when compared with businesses that received no program services. For businesses receiving SBDC assistance, the overall increase in median taxable income between 1999 and 2003 was approximately 150 percent. This compares with a slight decline in median taxable income for businesses, which did not access the program s services. Trends in Reported Employment Data We also analyzed business employment data from the Department of Labor and Industry (payroll and number of employees). Due to differences in availability of tax and employment data, employment Page 13

31 Chapter II Evaluation of SBDC Service Provision data comparison groups are slightly different from the taxable income groups discussed above. Only businesses reporting six years of payroll and employment data between 1999 and 2004 were included in this analysis. We calculated the median annual payroll and the median annual number of employees for businesses in the program and comparison groups. Between 1999 and 2004, businesses receiving assistance from the SBDC program increased their median payroll by around 90 percent. Over the same period, businesses that did not receive SBDC services increased median payroll by around 10 percent. When looking at the number of employees for comparison groups, trends show businesses receiving SBDC services added employees at a slower rate than businesses receiving no program services. Between 1999 and 2004, SBDC businesses increased median employment by five percent and non- SBDC businesses increased employment by around thirty-five percent. The following figure illustrates trends in employment data expressed as the median payroll per employee reported annually for SBDC and non-sbdc businesses between 1999 and Page 14

32 Chapter II Evaluation of SBDC Service Provision Figure 4 Median Annual Payroll per Employee For Comparison Groups Calendar Years $15,000 Payroll Per Employee $13,000 $11,000 $9,000 $7,000 $5, SBDC Businesses Non-SBDC Businesses Source: Compiled by the Legislative Audit Division from Department of Commerce and Department of Labor and Industry records, and other sources. Trends for Population Sub- Sectors Further analysis of sub-sectors of the population identified several noteworthy trends in both annual taxable income and annual payroll per employee: Time-Intensive Counseling businesses receiving more timeintensive SBDC counseling (defined as more than 10 hours), show enhanced benefits when compared against businesses receiving less time-intensive counseling. These issues are discussed further in chapter IV. Urban/Rural the program s services appear to have less impact for clients in rural areas as opposed to urban areas. Several factors could be important in explaining this trend, including the dynamics of rural economies, resources available to rural SBDC sub-centers, and access to other specialized service providers. New/Existing Businesses new or start-up businesses initially receive greater benefits through SBDC counseling than existing businesses. However, trends for new businesses are volatile and Page 15

33 Chapter II Evaluation of SBDC Service Provision it is unclear if the improvements would be sustainable over the long-term. Again, further discussion of these issues are contained in chapter IV. SBDC Business Benefit from Enhanced Productivity Overall, the comparison group study of SBDC businesses shows the program is delivering benefits to participants. Data from the SBDC client businesses included in our sample shows strong levels of growth in taxable income and payroll, and slight increases in the number of employees between 1999 and Evidence of a link between improved business performance and program services is provided by the comparison with a matched group of non-sbdc businesses. In terms of growth in annual taxable income and annual payroll per employee, participation in the SBDC program appears to provide small businesses in Montana with significant benefits, which they may not otherwise have experienced. The benefits provided by SBDC participation appear to be related primarily to improvements in business efficiency and productivity. Improvements are evident in taxable income and payroll per employee reported by SBDC businesses, but the program s counseling services do not appear to have a significant impact on the actual number of employees in client businesses. This effect is more notable for existing businesses, which constituted the majority of businesses in our comparison groups. Economic development programs are frequently portrayed as job creation mechanisms. However, this analysis shows that technical assistance programs are better understood as a means for creating value in client businesses through improvements in efficiency and productivity. SBDC Program Meets Its Primary Objective Although comparison group methodologies cannot control for all observable and unobservable variables affecting private sector businesses, it is reasonably likely that the identified improvements in business performance are attributable at least in part to SBDC services. The basic objective for the SBDC program is to provide guidance and advice for individual clients and improve their business performance. On this basis, the positive outcomes identified in the comparison group study justify the continuing provision of program services. Page 16

34 Chapter II Evaluation of SBDC Service Provision Conclusion: The SBDC program meets its primary objective of improving client business performance through the provision of counseling services. Small businesses seeking technical assistance also have access to other service providers in both the public and private sectors. The benefits provided by SBDC services are important, but the program must also be able to demonstrate that alternative programs or organizations cannot provide similar services and benefits or can do so at reduced cost. Assessments of SBDC service provision, private sector services, and other public sector programs are contained in the following sections. SBDC Service Provision SBDC counseling activities focus on providing small business people with advice and assistance on basic business planning functions. To better define the scope of activities for the program, we interviewed SBDC counselors and staff in the lead center and reviewed client files to determine areas of counseling. We identified three main areas where SBDC sub-centers provide assistance: Business Plan the core function of the program is providing advice and assistance in the development of business plans, and associated financial and market analysis necessary to complete the business planning process. Generally, this involves directing clients towards resources they can use in developing a business plan or reviewing and critiquing existing plans. Part of this function is the development and/or review of relatively simple financial projections for a business, including balance sheet and cash flow analysis. Financial Assistance a secondary SBDC service is helping clients identify and pursue available small business financing or loan programs. Financing and loan programs are available from private sector financial institutions, local revolving loan funds, and federal and state government loan programs. SBDC counselors are prohibited from involvement in loan decisions involving their clients, but they are encouraged to take an active interest in helping clients secure funding. Information Resource a third type of assistance provided by SBDC is acting as a information resource for the various public Page 17

35 Chapter II Evaluation of SBDC Service Provision and private sector resources and services available to small businesses. SBDC sub-centers are community-based and a common first contact for small business owners looking for information on available resources. Counselors are experienced in analyzing the specific needs of the small business community and referring clients to specialized services in both the public and private sectors. We obtained program data showing the type of counseling assistance provided to clients. SBDC counselors are required to define and record the type of counseling provided to each client in the program s management information system. We analyzed counseling type data from 1998 through The following table shows the proportions of different counseling types as defined by SBA reporting parameters. Table 4 Type of Counseling Services Provided Calendar Years SBA Counseling Area Percentage of Total Business Start-up/Acquisition 52.5 % Sources of Capital 20.8 % Government Procurement 9.0 % Financial Analysis/Cost Control 7.6 % Accounting & Records 4.0 % Marketing/Sales 3.8 % Business Liquidation/Sale 1.4 % Personnel 0.4 % Technology 0.2 % Inventory Control 0.1 % Engineering R&D 0.1 % International Trade 0.1 % Source: Compiled by the Legislative Audit Division from department records. Page 18 As shown, slightly over half of all SBDC counseling sessions focus on business start-up activities. Other categories reporting significant levels of activity are sources of capital, government procurement,

36 Chapter II Evaluation of SBDC Service Provision financial analysis/cost control, accounting and records, and marketing/sales. Program data for SBA counseling areas supports the views of the SBDC counselors and lead center staff regarding the program s role: the majority of counseling activities are focused on the initial stages of starting a business, including developing a business plan and the associated elements of the plan. Private Sector Service Providers To determine the availability of services in the private sector, we obtained data from the U.S Census Bureau from the 2002 Economic Census showing businesses providing management consulting services. In total, census data show there were 229 businesses in Montana offering management consulting services in Of these, 134 were categorized as providing administrative management and general management services ; individual business types within this sub-category include those providing services closely related to those offered by the SBDC program. Overall, approximately 80 percent of these businesses were located in seven counties in the state s main urban areas (Cascade, Flathead, Gallatin, Lewis & Clark, Missoula, Silver Bow, and Yellowstone.) Calculating the Cost of Private Services We obtained information from a national survey of firms in the management consulting sector. Information gathered in this survey showed the average hourly rate for entry-level analysts in management consulting firms is $147. Hourly rates rise with the level of experience and seniority of employees to over $300 per hour for senior partners in management consulting firms. To determine if private sector management consultants could provide services similar to SBDC at a comparable cost, we used data on average hourly rates for management consulting services and program statistics for counseling hours. Costs were calculated on the basis of cost per client (the billable amount a client would pay for services). Cost per client calculates the amount a small business owner would need to pay in fees to retain the services of a private sector management consultant. The average number of counseling hours Page 19

37 Chapter II Evaluation of SBDC Service Provision per SBDC client is around 4.5 hours. Based on hourly billable rate of $150, private sector management consultant services would cost $672 for the average client. The calculated cost should be considered the minimum, as it does not include travel hours and covers only the services of an entry-level analyst position in a management-consulting firm. Private Sector Services Are Available, But Costly In the absence of the SBDC program, the state s small businesses could turn to private sector management consultants to provide business technical assistance services. In this sense, program services compete with the private sector. However, many SBDC clients are start-up businesses in the early stages of development. The services typically being sought by these businesses are related to basic business planning functions. Private sector management consultants typically concentrate on more specialized services and offer assistance and advice relating to complex business processes, financial planning, and other specialized analysis. Even if private sector providers were able to readily offer services appropriate to the SBDC client base, at a minimum cost of $672, it is questionable whether many small businesses could afford to access these services. Given the types of services generally offered by management consultants and the costs involved in retaining the services of specialized firms, it is unlikely SBDC functions would be provided solely through the private sector. Discontinuing federal and state funding for SBDC services would be unlikely to result in the program s client base switching to private sector providers. Conclusion: Private sector providers could not function as the sole source for small business counseling services at a cost comparable to the SBDC program. Other Service Providers Business technical assistance is also provided by public sector organizations, public-private partnerships, and non-profit groups. There are multiple providers offering various forms of technical assistance. Some of these providers offer specialized services and others are more general in nature. It is important to note that not all Page 20

38 Chapter II Evaluation of SBDC Service Provision these organizations can or will provide services similar to those provided by SBDC sub-centers. Some providers have a specialized focus. Others may provide only basic services or access to information for small business owners (for example, local economic development corporations often focus on loan/financing activities or basic business advocacy functions, but may offer only limited technical assistance.) Multiple Programs Provide Business Technical Assistance Using input from SBDC program staff and background information on economic development programs in general, we identified the following programs or providers: Certified Regional Development Corporations CRDC program is administered by the Department of Commerce and consists of 11 economic development corporations working on a regional basis to build capacity and provide services to businesses. Most CRDC participants are also host agencies for the SBDC program. Regional Development Officers Department of Commerce RDOs are based in five communities around the state and provide a regional focus for department economic development activities and technical assistance on projects with a regional impact. Service Corps of Retired Executives The SCORE program uses the skills and expertise of retired business executives to assist businesses (particularly smaller businesses). SCORE is organized into eight local chapters and relies on volunteers to provide services. SCORE activities are supported by SBA. Montana Manufacturing Extension Center MMEC is a state and federally funded program providing advice and assistance for manufacturing and engineering businesses. The program has five locations around the state, including the main center at Montana State University-Bozeman. Procurement Technical Assistance Centers The PTAC program is administered by the U.S. Department of Defense and focuses Page 21

39 Chapter II Evaluation of SBDC Service Provision primarily on providing advice and assistance for businesses interested in bidding on contracts or being involved in projects with the federal government. The program has three locations in Montana. Resource Conservation & Development Organizations RC&D organizations are part of a program administered by the U.S. Department of Agriculture (USDA) offering support and technical assistance for rural economies. There are eight assigned RC&D areas around the state. USDA Rural Development The USDA Rural Development program offers multiple services to rural communities, including business-cooperative assistance. Program offices are located in six communities. Tribal Business Information Centers This program is also administered by SBA and provides business technical and other assistance to small businesses on Indian reservations. Two TBIC locations are currently functioning out of seven centers assigned by the SBA to state s tribal colleges. Business Incubators There are three business incubator programs in Montana. Incubators generally provide office space, other shared resources, and advice and assistance for start-up businesses. The three incubators identified in Montana are all members of the National Business Incubators Association. Local Economic Development Corporations The largest group of organizations providing business technical assistance are economic development groups established as private corporations. LEDCs are usually funded as private membership organizations, or partially by county or municipal governments. All the SBDC host agencies are also LEDCs. Page 22

40 Chapter II Evaluation of SBDC Service Provision Locations of Public Sector Providers Overall, these organizations have a presence in 46 Montana communities (larger communities have more than one of these service providers and in several cases a single organization is the location for multiple programs.) The following figure shows provider locations around the state. Page 23

41 Figure 5 46 Montana Communities With Business Technical Assistance Resources Source: Compiled by Legislative Audit Division from department records and other sources.

42 Chapter II Evaluation of SBD Service Provision Overlap Exists Among Service Providers The SBDC program is only one of multiple business technical assistance providers in the state. Review of information from other providers showed each program or service has a specific mission, but in some cases there is not always a clear boundary established between SBDC resources and services and those of other providers. Two examples of apparent duplication/overlap in service provision were identified: CRDC Program in many locations around the state, the SBDC host organization is also a certified technical assistance provider through the department s CRDC program. However, Flathead, Missoula, Ravalli, and Richland counties are not currently included in the CRDC program, but are included within SBDC service areas. For these areas, it is not clear how the delivery of business technical assistance is coordinated between different service providers. RDO Program RDOs are available as a resource for business location or expansion projects with a regional impact. Although SBDC clients are mainly small, start-up businesses, the program also counsels existing businesses with expansion plans, some of which could have significant regional impacts. It is unclear how the two programs are supposed to coordinate their activities with clients which are existing/established businesses. Although we could not identify specific examples of negative impacts on program efficiency or effectiveness, some level of confusion exists among SBDC counselors regarding the role of their program relative to other public sector providers. It should also be noted that a 1995 Legislative Audit Division performance audit of the RDO program identified concerns over coordination with other programs, including the SBDC program. The department s position is that a greater degree of coordination between the service areas of the SBDC and CRDC programs is a long-term goal in the development of a regional approach to economic development. In relation to SBDC coordination with the RDO program, the department maintains that the clienteles and size of project for the two programs are sufficiently different to avoid duplication of effort. Additionally, the department believes a good cooperative Page 25

43 Chapter II Evaluation of SBDC Service Provision relationship exists between the two programs and that the efforts of both programs effectively complement each other. Conclusion: Although the SBDC program provides distinct services, multiple providers within the public sector offer business technical assistance and this increases duplication of effort where programs and resources are not effectively coordinated. Overall Conclusion Results from our comparison group study and analysis of alternative service providers shows provision of SBDC services is beneficial for the state s small businesses. There is a demonstrated impact on the business productivity of participants in the program, and SBDC services are not readily available at a comparable cost in the private sector. There are some reasons to be concerned about the number of business technical assistance providers and duplication in services offered by these programs. However, despite overlaps in service provision, the SBDC program occupies a niche position and its services are not entirely replicated by any other provider. SBDC Program Could Improve Operational Efficiency and Effectiveness Further audit work addressed the operational efficiency and effectiveness of Montana s SBDC program. The following chapters include findings, conclusions and recommendations addressing further improvements in program operations. Identified improvements should allow the program to enhance its ability to effectively serve the state s small business community. Page 26

44 Chapter III - Program Compliance & Administration Introduction The administration of SBDC networks by states must meet compliance standards defined by the federal government. Interactions between the federal SBA and state entities administering SBDC programs are governed by the terms of cooperative agreements. SBA administrative oversight of SBDC operations also includes regular financial audits and programmatic reviews. Audit objectives addressed the extent to which the SBDC program meets federal compliance standards and the degree of consistency between Montana s administrative structure and programs in other states. Federal Administrative & Compliance Standards The Small Business Act does not specifically address the administrative role of the state lead center. Compliance standards outlined in the act are restricted to the role of the lead center as a program applicant and contract party with the SBA. In relation to the administration of the SBDC program, the Code of Federal Regulations is the primary compliance reference at the federal level. Federal regulations provide general guidelines rather than specific directives. Program Meets Federal Compliance Standards To determine if the lead center is performing its duties in compliance with federal laws and regulations, we reviewed available documentation and interviewed staff in the lead center and SBDC counselors working in sub-centers. At a basic level, the SBDC program can be said to meet compliance standards because the federal government has continued to renew the program s cooperative agreement in successive years. Under federal regulations, the SBA Office of Small Business Development Centers has the ability to take corrective actions against state networks that are not complying with laws and regulations applicable to the program. In cases where non-compliance is identified, cooperative agreements with the recipient organization can be terminated or granted on a provisional basis. Additionally, review of specific compliance elements identified in federal law and regulations shows Montana s SBDC program meets Page 27

45 Chapter III Program Compliance & Administration general administrative standards. Review of lead center policies and procedures showed program management is actively addressing the following federal compliance standards: Provision and coordination of counseling and training services. Independent status of the lead center. Duties and responsibilities of the State Director. Maintenance of conflict of interest policies for SBDC staff and counselors. Counselor involvement in loan or financing decisions for clients. Implementation of special emphasis initiatives for specific client groups. Establishment of program advisory boards. Program Oversight Procedures Under oversight regulations established by the federal SBA, lead centers are required to conduct regular inspections of SBDC subcenters (including site visits to sub-centers). To assess the effectiveness of the oversight procedures followed by the SBDC lead center, we reviewed guidelines issued by the SBA, obtained and reviewed examples of three different oversight inspection reports compiled by lead center staff, and discussed oversight procedures with lead center staff and counselors working in SBDC sub-centers. In addition, we reviewed documents relating to the ASBDC accreditation process for state SBDC networks. Contractual Relationship Between Lead and Sub- Centers The ability of the lead center to exercise effective oversight of subcenters is impacted by the program s management control structure. SBDC counselors are employees of host agencies and are not subject to direct supervision by the lead center (as department employees would be.) The remote nature of the relationship with sub-centers can pose problems for the lead center. However, conduct or performance issues with SBDC counselors can be addressed through enforcement of contractual terms. The department retains the options of either termination of contracts for services or, through the Request for Proposal (RFP) process, additions or refinements to contractual terms for particular sub-centers. Page 28

46 Chapter III Program Compliance & Administration Lead Center Meets Federal Compliance and Oversight Standards From the perspective of compliance with federal regulations, the lead center maintains required policies and procedures, and meets other management control standards. Lead center staff conducts on-site inspections and other reviews as required. Lead center oversight activities follow a regular schedule, include all sub-centers at least once per year, and address all the oversight elements prescribed by the SBA. Conclusion: The lead center performs its duties in accordance with the general administrative compliance and oversight standards outlined in federal laws and regulations. SBA Business Size Regulations In common with all SBA-administered programs, SBDC operations are also governed by federal regulations relating to the size of the businesses receiving assistance. Title 13, Part 121, CFR, determines business size thresholds either according to the number of employees or the total receipts of businesses. These levels vary according to the industry sector the business operates in. For example, a business operating in the logging sector (forest products) must have fewer than 500 employees to be eligible for SBA program assistance. These guidelines ensure only small-sized businesses receive SBDC services. Trends in Client Business Types Demographic analysis of SBDC participants showed significant trends could be impacting program compliance with SBA business size regulations. Analysis of client business types addressed business status (new or existing business), organizational type (form of legal ownership), and number of employees. Trends in these areas are discussed as follows: Business Status - in 1998, 29 percent of SBDC clients were existing businesses and 71 percent were start-ups. In 2004, 46 percent of program participants were existing businesses and 54 percent were start-ups. This trend corresponds with SBA efforts to focus on providing more services to existing businesses. Organizational Type - analysis of business organizational types showed the percentage of SBDC businesses organized as sole Page 29

47 Chapter III Program Compliance & Administration proprietorships or partnerships is declining, while the percentage formally organized as corporations, sub-s corporations, or LLCs is increasing. Number of Employees - program data shows that on average SBDC businesses have 3.3 full-time employees. Between 1998 and 2004 there has been an increase in the average number of full-time employees for SBDC businesses. Again, this trend is likely related to the increasing proportion of existing businesses using the program. As the program matures, participation has shifted towards more established existing businesses with more employees and more complex organizational structures. A declining proportion of SBDC clients are one-person, micro-businesses in the very early stages of development. As these trends have become evident, the potential for non-compliance with SBA business size regulations has increased. Some Businesses Exceed Size Regulations To determine whether businesses participating in the Montana SBDC program meet size regulations, we reviewed records for 15 selected businesses with more than 100 employees. All these businesses received SBDC services between 1998 and Eight of the fifteen businesses included in our review did not meet SBA business size regulations, either because they had too many employees or their annual receipts exceeded allowed levels. For these eight businesses, the program provided around 68 hours of counseling services. For three of the businesses reviewed, we were unable to determine whether they exceeded business size regulations due to a lack of data on their annual receipts. The remaining four businesses complied with SBA business size regulations. SBDC Over-Reported Some Client Data In addition to non-compliance with federal regulations, the SBDC program has over-reported both the number of clients served by the program, and outcomes associated with these clients. For the eight businesses we identified as exceeding SBA size regulations, subcenters have recorded the following outcomes: 1,515 jobs retained 69 jobs created Page 30 $12,877,000 investments

48 Chapter III Program Compliance & Administration $254,110,000 project size Because the businesses were not eligible to receive SBDC assistance, the reported outputs and outcomes associated with them should not be used for the purposes of assessing program performance and reporting to the legislature and other funding entities. Program Managers Need to Improve Oversight of Client Records SBDC host agencies are often involved in expansion or recruitment projects for large businesses and SBDC counselors can be involved in these activities. In conducting these activities, SBDC counselors need to be aware that these businesses often do not meet size regulations established by the federal government. Counselors also need to be aware that businesses in some industry sectors (especially technology and business/technical services), are defined according to their annual receipts; while these companies may not have large numbers of employees, they could still exceed the thresholds established by the SBA. Recording these activities in the SBDC database can result in over-reporting of the total number of clients being served by the program and in the program s reported outcomes. Improving awareness of the SBA business size regulations among sub-center directors could prevent future occurrences of noncomplying businesses accessing SBDC services. Improving lead center oversight of sub-center data reporting could help prevent overreporting of SBDC outcomes. Oversight procedures need not be overly burdensome as companies likely to exceed SBA business size regulations are generally larger or operate in specific industry sectors. The lead center could improve controls by selectively identifying certain companies for additional review as part of normal quarterly or annual reporting procedures. Additional review could include further research to determine the client s size. Page 31

49 Chapter III Program Compliance & Administration Recommendation #1 We recommend the department improve review of SBDC client records to ensure all participating companies are eligible under SBA business size regulations and program outcomes are reported accurately. Comparison with Other States All 50 states currently participate in the SBDC program and there are over 1,100 sub-centers in locations around the country. Audit objectives included analysis of Montana s SBDC program as compared with other states to assess the efficiency and effectiveness of the structure and organization of the network. Analysis addressed multiple issues, including the role of universities in administering the program, network/sub-center structures, and federal and state funding of SBDC networks. University System Administration of SBDC Programs The most prominent difference between Montana s SBDC program and those in most other states is administration of the network outside of the university system. All but seven participating states administer the SBDC program through university systems, using universities and community colleges as host agencies for sub-centers. Alternative models are used in Montana, Colorado, Illinois, Indiana, Minnesota, Ohio, and West Virginia. The alternative model uses non-university organizations or agencies as lead centers. Federal regulations provide exceptions for states using non-university system host agencies, but any future SBDC lead centers in these states will be required to be located within university systems. The Department of Commerce notes the current approach in Montana uses local economic development organizations as host agencies and co-locates SBDC services with other small business resources, including local revolving loan funds. Network Structures in Western States We reviewed the network structures for SBDC programs in ten western states to determine if the number and distribution of subcenter locations in Montana is consistent with other states. Analysis addressed the average population base per sub-center (as a measure of the potential clientele), and the average land area covered by each Page 32

50 Chapter III Program Compliance & Administration sub-center. Information was sourced from lead centers in other states and is presented in the following table. Table 5 SBDC Network Structures in Montana and Other Western States SBDC Locations Average Sub-Center Population Base Average Sub-Center Land Area (square miles) Colorado 21 Arizona 513,063 Wyoming 19,564 New Mexico 18 Idaho 215,659 Montana 14,705 Nevada 11 Colorado 204,822 Idaho 13,929 Utah 11 Utah 203,015 South Dakota 12,854 Arizona 10 Nevada 181,660 North Dakota 11,784 Montana 10 South Dakota 125,807 Arizona 11,401 Idaho 6 North Dakota 107,033 Nevada 10,052 North Dakota 6 New Mexico 101,058 Utah 7,719 South Dakota 6 Wyoming 98,756 New Mexico 6,755 Wyoming 5 Montana 90,220 Colorado 4,957 Source: Compiled by Legislative Audit Division from department records and information from other states. States with larger populations tend to have more SBDC locations, and smaller-population states have fewer. The number of subcenters in Montana is inconsistent with this general trend. The explanation for this inconsistency is apparent in the population and land area data. Montana s large land area and relatively low population density result in additional sub-center locations as compared with other states. Montana s SBDC sub-centers serve the smallest average population bases among western states, but are also required to cover the second largest average land area. By increasing distances between clients and sub-centers, reducing the number of sub-center locations could increase the average land area beyond levels where SBDC counselors could provide services effectively. Conclusion: Montana s SBDC network structure is consistent with the state s low population density and high land area as compared with other western states. Page 33

51 Chapter III Program Compliance & Administration Federal and State Funding Federal law establishes the distribution of funding to state SBDC programs and also sets minimum matching requirements for recipient organizations. However, states are accorded wide latitude in identifying sources of funding for the program. Funding data for states participating in the SBDC program was obtained from the ASBDC and is based on a survey conducted by program participants. To determine how SBDC funding in Montana compares with other states, we analyzed data for states receiving at or near the minimum federal funding (approximately $500,000). Comparison of funding levels in these states included calculations of state funding amounts as a percentage of the federal appropriation (to reflect the extent of state commitment to the program in funding terms.) We also included a calculated estimate for the amount of funds from nonfederal or state government sources. Results are shown in the following table. State Table 6 State and Non-Governmental Funding of SBDC Programs Calendar Years 2005 State Funds % of Federal Funds State Approximate Funds from Non-Governmental Sources ** New Mexico 838 % Alaska $2,500,000 West Virginia 387 % Maine $1,000,000 Maine 146 % West Virginia $800,000 Nebraska 117 % Montana $650,000 Alaska 110 % Delaware $500,000 Delaware 85 % Idaho $500,000 South Dakota 70 % New Mexico $400,000 Idaho 58 % North Dakota $400,000 New Hampshire 56 % South Dakota $300,000 Montana * 50 % New Hampshire $300,000 Rhode Island 35 % Nebraska $150,000 North Dakota 33 % Rhode Island $75,000 * Montana State-sourced funds include General Fund and statutory appropriations to the SBDC program. ** Estimate based on values projected from data supplied in the state funding survey. May not reflect actual funding level. Source: Compiled by the Legislative Audit Division from Association of Small Business Development Centers records. Page 34

52 Chapter III Program Compliance & Administration Montana contributes state funding to the SBDC program at a rate of 50 percent of federal funds. This is not as high as some comparable states, but the level at which Montana funds the program through other sources is higher than most other states. In this context, Montana is effectively using alternative funding sources (in this case, funds from local economic development organizations) to leverage state funding for the program. Analysis of overall funding levels shows Montana s program has more available resources than the majority of other states receiving the minimum federal appropriation. Conclusion: When compared with other states, Montana has been effective in using host agency contributions to leverage state funding and increase resources available for SBDC program operations. Overall Conclusion The department s administration of the SBDC program in Montana is meeting general federal compliance standards and is consistent with other states. As the program s clientele changes, improvements will be necessary in review procedures to ensure participating businesses comply with SBA business size regulations. In comparison with other states, Montana s program is notably different in its use of local economic development corporations to deliver services. In other respects, Montana s administrative approach is consistent with other states around the country. Page 35

53 Page 36

54 Chapter IV Counseling & Training Activities Introduction SBDC counselors provide two primary services to clients; individual counseling sessions and group training events. Conducting and managing these activities represents the largest commitment of time and resources made by the program. Audit objectives addressed assessment of SBDC service provision to determine how well the program serves the state s small business community. Methodologies included review of client and training files in subcenters, interviews with counselors and program staff, and analysis of program data. This chapter addresses findings and conclusions in relation to trends in counseling and training activities, and includes recommendations to improve the delivery and management of program services. Data Analysis Definitions and Parameters The data presented in the following sections was sourced from the SBDC management information system. Unless otherwise stated, analysis covers calendar years 1998 to Data includes recorded activities for the program s ten sub-centers and does not include data recorded by the Helena lead center. For the purposes of our analysis, a counseling session was defined as any direct contact with a client lasting one half hour or more. A training event was defined as any SBDC-sponsored event with recorded attendance of at least one individual. Sub-Center Documentation Procedures SBDC sub-centers are required under federal regulations to maintain files documenting counseling and training services provided to clients. We reviewed counseling and training files to ensure subcenter documentation procedures provide sufficient support for reported client and training activity records. We reviewed client and training files from five selected sub-centers. Files for 62 individual clients and 26 training events from fiscal year 2004 were included for review. Counseling & Training Activities are Documented Correctly Review of files in selected SBDC sub-centers showed documentation standards for counseling and training activities are generally good. We did not identify any significant problems with the availability, completeness or accuracy of counseling and training files. SBDC Page 37

55 Chapter IV Counseling & Training Activities sub-centers meet documentation requirements established by the SBA, and the majority of files contained the required federal reporting forms. Reported levels of client counseling and training event activities were accurate and can be reliably used as program output indicators. Conclusion: SBDC counseling and training activities are correctly documented by sub-center staff and reported data for these activities are accurate. Counseling and Training Activity Trends The number of clients counseled is the most basic activity or output measure available for the SBDC program. Analysis of trend data for clients counseled between 1998 and 2004 showed that there has been a gradual decline in the number of clients counseled by the program. As the number of clients counseled has decreased, the average number of counseling hours per client has increased. This trend is illustrated in the following figure. Figure 6 SBDC Clients Counseled and Average Hours Per Client Calendar Year Clients 1,800 1,600 1,400 1,200 1, Hours Clients Counseled (left axis) Hours Per Client (right axis) Source: Compiled by the Legislative Audit Division from department records. Page 38

56 Chapter IV Counseling & Training Activities Number of Clients Counseled is Decreasing For all sub-centers, there is a trend towards clients receiving more time-intensive counseling (as measured by the average number of hours per client). According to lead center staff, this trend could be related to efforts at increasing the quality of counseling available to clients. As evidenced by the analyzed counseling data, the program s attempts to promote more time-intensive counseling appear to be working. Training Events and Attendance are Increasing Organizing and teaching/leading training events is the second primary activity for SBDC counselors. Training events are generally held in the host agency s facilities or in satellite locations around the service area. We analyzed training data for 1998 through 2004 to identify how many events were held by each sub-center in each year. Additional analysis addressed the number of training hours recorded and the number of attendees for events. There are clear trends evident in training data indicating the number of events is increasing across the network. As a result, the number of training hours and attendees is also increasing. Between 1998 and 2004, the average annual growth in training events was 11 percent and the average annual growth in training attendance was 9 percent. Program Uses Training to Screen Participants Discussions with SBDC counselors and lead center staff indicate the program is using training events to screen for participants with more advanced business plans. For individuals with no business plan, oneon-one SBDC counseling may not be an efficient use of resources, as the counselor would not have access to the materials necessary to evaluate a business idea (business plan, financial projections, market analysis and others). By channeling individuals into Business 101 training events, counselors can provide basic counseling to multiple individuals at the same time. The use of training events to screen program applicants for suitability appears to be having an effect on the program s counseling activities. As client workloads decline, counselors can spend more time counseling individual. The effects of this approach are initially being seen in the increasing average number of hours per client recorded by SBDC sub-centers. Page 39

57 Chapter IV Counseling & Training Activities Conclusion: The Montana SBDC program is pursuing a strategy of reducing client workloads through time-intensive counseling and expanded training opportunities. Analysis of Training Event Locations Additional analysis of training addressed locations of events in cities and towns around the state. A total of 62 different communities around the state were locations for SBDC training. The vast majority of training events are held in cities where a SBDC subcenter is located. Both urban and rural locations were chosen for events, but the majority are held in the state s larger urban areas. We compared training locations with sub-center service areas to determine how many locations were used by each sub-center. In general, sub-centers ensured training events were held in different communities around the service area. The exception to this trend was the Billings sub-center. Training records show the Billings SBDC conducted training workshops in three communities within its service area (Billings, Laurel and Crow Agency.) Under the terms of the program s cooperative agreement, the Billings sub-center is to serve a nine-county area, including communities outside Yellowstone county. Sub-centers located in other urban areas of the state have ensured training events are not concentrated solely in the service area s main city. For example, the Missoula sub-center also serves a large urban area, but sponsored training events in smaller communities in rural counties in the west and north of its service area. Program Should Address Availability of Training in Some Locations In the absence of specific direction through cooperative agreements or contracts, the Billings sub-center concentrated training activities in Billings based on the assumption that clients from outlying communities can travel to the city to attend events. The Billings subcenter could more effectively serve the population of its service area by sponsoring training events in smaller communities such as Big Timber, Columbus, Red Lodge and Harlowton. Updates to cooperative agreements and host agency contracts should include reference to outreach and, particularly, training opportunities in rural communities within all SBDC service areas. Page 40

58 Chapter IV Counseling & Training Activities Recommendation #2 We recommend the department ensure cooperative agreements and host agency contracts address provision of training opportunities in outlying communities within SBDC service areas. Client Demographic Trends The SBDC program collects demographic data for all clients receiving counseling. The demographic data collected cover multiple areas, including client gender, ethnicity, and veteran status. Demographic data for program participants between 1998 and 2004 was analyzed to identify trends at both the network and sub-center level. Analysis identified notable features in relation to participation by veterans and American Indians. These issues are discussed in the following sections. Special Emphasis Groups Under federal regulations the SBA can designate a particular group within a served population for increased emphasis. Known as special emphasis initiatives, these efforts are designed to target particular groups for increased outreach and promotion efforts and boost their rate of participation in the program. The following sections address two special emphasis groups currently being undertaken by Montana s SBDC program; veterans of the armed services and American Indians. Trends in Veteran Participation Outreach to veterans has been identified by SBA and the Montana SBDC program as a special emphasis initiative. In the 2005 cooperative agreements between SBA and the department, veterans have been identified for specific outreach activities. To determine how well the SBDC program serves Montana s veterans, we analyzed participation rates for this group. Overall, the SBDC program does comparatively well at targeting the veteran population; veterans make-up around 11 percent of the state s working-age population and around percent of program participants. However, some disparities were observed at the sub-center level through analysis of data for veteran populations Page 41

59 Chapter IV Counseling & Training Activities in specific counties. County-level veteran population data from the 2000 Census was used to project a target SBDC participation level for the county based on the state average participation rate. By comparing projected veteran participation levels with the average number of veterans participating annually, we calculated a variance for each county. County data was aggregated to show the variation in veteran participation for each SBDC sub-center. Results are displayed in the following table. Table 7 Veteran Participation By SBDC Sub-Center Calendar Years Sub-Center Projected Veteran Clients Actual Veterans Served Veterans Served +/- Projection Colstrip SBDC Helena SBDC Havre SBDC Butte SBDC Great Falls SBDC Bozeman SBDC Wolf Point SBDC Missoula SBDC Kalispell SBDC Billings SBDC Source: Compiled by the Legislative Audit Division from department records and U.S. Census Bureau records. Three out of ten sub-centers are serving fewer veterans than the projected participation level; Billings, Kalispell and Missoula. Remaining sub-centers are generally exceeding projected participation levels, by significant amounts in some cases. It is not immediately clear why veterans in some service areas are failing to access SBDC services at the expected rate. It is possible SBDC services are not being utilized by veterans in these areas because of strong employment demand or other local factors. Page 42

60 Chapter IV Counseling & Training Activities Program Should Conduct Further Analysis of Veteran Participation For the program to effectively target veterans through outreach and other activities, it is necessary for managers to have detailed information on participation rates for the target group. Without monitoring and analysis of outreach efforts in different service areas, program managers cannot determine if the program is effective or succeeding in attempts to strengthen veteran participation. Program managers should continue to monitor the effectiveness of outreach efforts targeting the state s military veterans. In particular, the effectiveness of outreach efforts in larger cities should be the subject of more detailed analysis to determine if the veteran population in these areas is being effectively served by the program. Recommendation #3 We recommend the department perform analysis and other testing of veteran outreach efforts in the state s larger urban centers to ensure these populations are being effectively served by the SBDC program. American Indian Participation The other special emphasis initiative group targeted by Montana s SBDC program is the American Indian population. During interviews with SBDC counselors, concerns were identified over the program s effectiveness on the state s Indian reservations. In general, SBDC counselors are frustrated with an apparent lack of progress in extending the program s services to the reservations, and confused over how the program is supposed to coordinate activities with other service providers. Analysis of program participation data suggests the concerns of some of the SBDC counselors we spoke with are not misplaced. Since 1998, the absolute number of SBDC participants living on Indian reservations has been declining at a faster rate than the population as a whole. Between 1998 and 2004, SBDC participation has declined by around 3 percent annually, but for residents of the state s Indian reservations, the decline is closer to 6 percent. Page 43

61 Chapter IV Counseling & Training Activities Reservation Training Activities We examined the number and frequency of SBDC training activities provided on the state s Indian reservations. Review of training records for 1998 to 2004 showed that efforts on some of the state s reservations have been inconsistent. The following table shows the number of training events held on each reservation between 1998 and Table 8 SBDC Training Events on Indian Reservations Calendar Years Reservation Annual Training Events Blackfeet Crow Flathead Fort Belknap Fort Peck Northern Cheyenne Rocky Boy s Annual Total Source: Compiled by the Legislative Audit Division from department records. Program data reveals inconsistency in delivery of SBDC training services on the state s Indian reservations. Since 1998, the Flathead reservation has been the only reservation to consistently receive SBDC-sponsored training events. Declining levels of SBDC participation on Indian reservations may be attributable to generally poor economic conditions prevailing in some of these areas. While low levels of business activity may be affecting SBDC service delivery on reservations, greater consistency in the number and frequency of reservation training events could allow counselors to develop a higher profile for the program. Program management has recognized these trends and has mandated a minimum of two training events for each Indian reservation for the current program year. Page 44

62 Chapter IV Counseling & Training Activities Indian Economic Development Initiatives The SBDC program s strategic planning process has identified the state s American Indian population as a priority for outreach and promotion efforts. This effort mirrors other initiatives aimed at addressing economic development on the state s Indian reservations. The Department of Commerce is currently involved in cooperative efforts with the state s Indian tribes to identify and prioritize economic development initiatives on reservations. SBDC program services could, potentially, provide a valuable resource for tribal governments addressing the provision of business technical assistance services. Discussions with department management indicated there is currently no clear role for SBDC services and resources in these ongoing efforts. Addressing how SBDC services could be coordinated with other service providers or resources on reservations could help program managers meet their strategic goals more effectively. SBDC Program Should Address Reservation Activities The success of the SBDC special emphasis initiative relating to American Indians depends on program managers committing resources to outreach on reservations and ensuring services are effectively coordinated with other organizations. By ensuring an enhanced training schedule is properly implemented for Indian reservations, the SBDC sub-centers may be able to raise awareness of the services and resources available through the program. Additionally, addressing the place of SBDC resources in ongoing efforts to promote Indian economic development could help clarify the program s role on reservations and relations with tribal and other technical assistance providers. Recommendation #4 We recommend the department address SBDC participation on the state s Indian reservations by: A. Implementing a regular schedule of training events on reservations over a sustained timeframe; and B. Clarifying the role of SBDC technical assistance services in ongoing efforts to promote Indian economic development. Page 45

63 Chapter IV Counseling & Training Activities Overall Conclusion Trends in SBDC participation data indicate the program is changing its approach to managing client workloads. Counselors are using training events to screen potential participants and concentrate on viable business ideas. The result has been a gradual decrease in client workloads and an increase in the average number of hours of counseling per client. Essentially, the program is trying to emphasize quality of the counseling service over quantity in client workloads. As shown in the comparison group study discussed in chapter II, the move towards intensive counseling could be providing SBDC clients with significant improvements in business productivity. Our analysis of client participation and demographic data shows the program could also make improvements in several areas. Recommendations address improvements in training availability, veteran outreach efforts and program services on the state s Indian reservations. Page 46

64 Chapter V - Economic Trends & SBDC Service Areas Introduction The SBDC network is configured to deliver services to a specific population; the state s small business community. Although Montana is home to some larger businesses or subsidiaries of national and multi-national companies, the state s economy consists primarily of small businesses. As in any economy, patterns of business growth and investment are dynamic and subject to constant change as markets allocate resources. Our audit objectives included assessment of how closely the structure and organization of the SBDC network corresponds with the dynamics of the private sector economy. Specifically, we addressed the extent to which the current SBDC service areas are meeting the needs of the state s small business community and whether changes in the private sector economy are being reflected in allocation of program resources. These issues and additional analysis of SBDC client locations are discussed in the following sections. SBDC Client Locations Analysis of client locations between 1998 and 2004 showed the majority of SBDC clients are from the state s main urban centers. We analyzed zip code data for SBDC clients in order to map the distribution of the client base relative to service areas and subcenters. The following figure depicts the SBDC client base using a dot density distribution. Page 47

65 Chapter V Economic Trends & SBDC Service Areas Figure 7 Density Distribution of SBDC Clients Calendar Years Source: Compiled by the Legislative Audit Division from department records. SBDC Serves a Predominately Urban Clientele As shown, the program serves a predominately urban clientele, with the main concentrations of participants found in the state s larger cities and towns. In this respect, the program s resources are distributed in line with what should be expected based on the distribution of the state s population, which is concentrated in urban areas. Further analysis calculated the proportions of SBDC clients from either urban or rural locations. Overall, 61 percent of program participants are from urban communities and 39 percent from rural. Page 48 Additional analysis of client locations addressed the number of clients from other states receiving SBDC services. Federal regulations allow SBDC programs to serve clients from outside their state jurisdiction (where access to a sub-center is made easier by crossing state lines). Additionally, the program provides counseling services to individuals from other states considering locating/relocating a business in the state. Analysis of address details supplied by program participants showed around 220 clients identified themselves as non-residents of the state between 1998 and This is about 2 percent of the program s total client base.

66 Chapter V Economic Trends & SBDC Service Areas Small Business Activity in SBDC Service Areas To analyze trends in small business activity within SBDC service areas, we obtained data from the U.S. Census Bureau showing the growth in the number of business establishments at the county level. The Census Bureau publishes two annual data series used in this analysis; the County Business Patterns data series covering businesses with paid employees, and the Non-Employer Statistics series, which includes data for businesses without paid employees (sole proprietorship businesses owned and operated by the selfemployed). Both these data series provide information on the number of business establishments in each Montana county within all industry sectors. Analysis of County Economic Growth Trends Data from the County Business Patterns and Non-Employer Statistics series were analyzed to identify growth trends in the number of business establishments for all counties between 1997 and Overall, this data showed the average annual growth in the number of business establishments for the state was 1.58 percent. As expected, significant disparities existed in trends for different counties. Economic growth in the state has not been even in recent years and counties experiencing significant growth rates are concentrated in the south and west of Montana. County growth trend data was aggregated based on the SBDC service areas to show overall trends affecting each of the program s sub-centers. The following figure illustrates growth trends for each SBDC service area. Page 49

67 Chapter V Economic Trends & SBDC Service Areas Figure 8 Growth in Business Establishments by SBDC Service Area Calendar Years Source: Compiled by the Legislative Audit Division from U.S. Census Bureau data. SBDC service areas have experienced varying growth rates in the number of business establishments; the Wolf Point, Colstrip, Havre, and Great Falls service areas have experienced negative (declining) growth rates; the Billings, Butte, and Helena service areas have seen moderate growth rates; and the Kalispell, Missoula and Bozeman service areas experience strong growth rates. Trends in SBDC Client Activities Page 50 Following analysis of economic trends, we reviewed client activity data for SBDC sub-centers to determine how well the network s structure corresponds with prevailing economic conditions. This analysis used the number of clients counseled annually in each subcenter and compared this with the overall client base of the service area (as measured by the total number of business establishments). We used business establishment data and SBDC client workloads to calculate a projected annual number of clients for each sub-center. The projected values for client workload represent an ideal

68 Chapter V Economic Trends & SBDC Service Areas distribution of SBDC workload between different sub-centers based on prevailing economic trends within the state. Although real-world conditions will never entirely replicate ideal-type distributions, analysis of the overall trends identified should allow us to draw some conclusions regarding the extent to which the SBDC network reflects prevailing economic conditions within the state. Observed Trends in Sub- Center Performance To analyze the trends in sub-center performance over time, we calculated the difference (variance) between actual sub-center workload and the projected client base (based on the level of economic activity in the service area). This variance value provides an approximate measure of how each SBDC sub-center performed above or below an ideal workload level. The following figure shows the median annual variance value over the whole time period for each sub-center. Figure 9 Median Annual Variance in Sub-Center Client Activity Performance Level Calendar Years Source: Compiled by the Legislative Audit Division from department records Page 51

69 Chapter V Economic Trends & SBDC Service Areas Three sub-centers (Missoula, Kalispell and Billings) have performed below the ideal level of client activity based on economic trends within their service areas. These sub-centers have served fewer clients than would otherwise be expected based on the projected levels of activity resulting from economic trends in their service areas. Remaining urban sub-centers (Bozeman, Helena, Butte and Great Falls) have generally performed above their ideal level of client activity. The three sub-centers serving predominately rural areas (Havre, Colstrip and Wolf Point), have continued to perform above their ideal client activity levels. Several Factors Could Affect Sub-Center Workload The observed trends in SBDC sub-center performance can be explained by the interplay between supply and demand factors within local economies. Specifically, four interrelated factors could be affecting SBDC sub-center workload levels: 1) Demand for SBDC services intensifying in urban areas in line with economic trends, demand for SBDC services is intensifying in the state s fast-growing urban areas and static or declining in rural areas. In three urban services areas (Missoula, Kalispell and Billings), SBDC sub-centers have not been able to meet demand for services as more businesses start-up or relocate in these areas. 2) Economic change in rural areas maintains SBDC activity levels demand for SBDC services appears to be fairly constant in three sub-centers located in predominately rural areas (Havre, Colstrip and Wolf Point). The declining viability of traditional industries and agriculture in rural areas could be helping to keep demand for SBDC services high as individuals explore selfemployment options in new business sectors. 3) Alternative service providers reduce demand for SBDC services the presence of alternative public sector economic development service providers (see Figure 5, Chapter II), could result in lower demand for SBDC services in urban areas. Prospective SBDC clients in urban areas could be accessing services from local economic development organizations, SCORE chapters or other providers. The relative scarcity of these providers in rural areas may explain why demand for SBDC services is consistently higher than would otherwise be expected. Page 52

70 Chapter V Economic Trends & SBDC Service Areas 4) Differing levels of success in implementing training programs certain sub-centers may be experiencing more success than others in implementing training programs to screen clients (see Chapter IV.) The Missoula, Kalispell and Billings sub-centers all have very active training programs and this may be helping these sub-centers screen clients more effectively and reduce their overall client activity levels. Program Should Address Resource Allocation As change in the wider economy accelerates, resource allocation within the SBDC network will become increasingly difficult to manage. Economic growth is intensifying in the state s urban centers and, at the same time, rural economies are contracting. For a program with a mandate to provide statewide access to services, SBDC faces the challenge of adjusting resource allocation to meet demand, while continuing to ensure services are provided to all Montana communities. Effects of Some Programmatic Trends are Uncertain Currently, it is unclear how the trends discussed above will impact SBDC client activity levels. Particularly, the effects of increased emphasis on training activities to screen clients is not being consistently demonstrated across the network. Some sub-centers are implementing active training programs and these are reducing their client activity levels, while sub-centers with less active training efforts are maintaining higher client bases than would otherwise be expected given prevailing economic conditions. Also, the presence of alternative service providers within particular service areas and relations between SBDC sub-centers and these providers could be impacting client activity levels. Program managers will need to continue to monitor implementation of training programs and client workloads in sub-centers, until a baseline level of client activity is established. Further Monitoring and Resource Allocation Changes are Necessary In relation to the administration of the SBDC program, the Code of Federal Regulations, CFR (a), states The SBDC network must provide prospective and existing small business persons and entities with counseling, training and specialized services, concerning the formation, financing, management and operation of small business enterprises, reflecting local needs Services Page 53

71 Chapter V Economic Trends & SBDC Service Areas periodically should be assessed and improved to keep pace with changing small business needs. The department should continue to monitor implementation of SBDC training programs in different sub-centers to determine how these activities impact overall client activity levels. Until a baseline level of client activity is established, program managers will need to continue to monitor implementation of training programs and client workloads in sub-centers. Monitoring should include assessment of other relevant output and outcome performance measures to ensure workload efficiencies are being maximized across the network. Client Activity Trends Not Commensurate With Prevailing Economic Conditions Despite some uncertainty regarding specific trends, it is evident that overall client activity levels for SBDC sub-centers are not commensurate with trends in economic activity within different service areas. As measured against a projected client base, SBDC sub-centers in fast-growth urban areas are failing to maintain levels of activity sufficient to meet demand for services. Given current population and economic trends in the state s fastest growing urban areas, it can be expected that the trends in SBDC activity levels will accelerate. Whereas demand can continue to be met in slow or nogrowth rural areas, SBDC sub-centers in some urban areas will increasingly face difficulties in serving all prospective clients. Intervention Required to Correct Resource Allocation Regardless of results from continuing monitoring of client activity levels, long-term trends point to a systemic imbalance in the program s allocation of resources between service areas. In terms of sub-center service areas, there have been no changes in the SBDC network structure since the program was established in the 1980s. Adjustments in the department s resource allocation methods have also been minimal. SBDC funding is distributed to sub-centers on an equal share basis (with the exception of additional funding for the Billings sub-center), rather than proportionally on the basis of workload or service demand. To avoid difficulties in serving clients in the state s urban areas, the department needs to adjust funding distribution and allocate Page 54

72 Chapter V Economic Trends & SBDC Service Areas resources to SBDC sub-centers based on prevailing economic trends within service areas. Department methodologies for allocating resources need to be adjusted over time to reflect economic reality. A number of options exist to effect these changes, including redistributing existing funding based on market demand, redirecting department resources to provide support for the SBDC network (for example, directing Regional Development Officers to support SBDC operations in rural areas), cooperating with other providers to support SBDC operations, and changing existing service area boundaries to reduce the client base in urban service areas. The department needs to use resources where they are needed most. The delivering of program services can be improved by allocating funding and other resources based on demand. Recommendation #5 We recommend the department ensure SBDC program administration serves the needs of the state s small businesses by: A. Monitoring implementation of training programs and continuing trends in client activity levels; and B. Adjusting funding and allocating resources to reflect demand for program services. Overall Conclusion The SBDC program was established to serve the state s small business community. As a result, adjustments in program strategies will always be necessary to reflect changes in the economy. The program is currently facing challenges imposed by shifts in the relative economic fortunes of different parts of the state. These changes are occurring in conjunction with changes in the program s approach to managing client workloads. Over time, adjustments in resource allocation methods should help the SBDC program continue to provide effective services for all the state s small businesses. Page 55

73 Page 56

74 Chapter VI Program Outcomes Reporting Introduction In common with most economic development functions, the SBDC program seeks to quantify the economic impact of its services. The program uses a performance measurement system to track both outputs (measures of program activity, such as the number of clients served), and outcomes (measures of the program s effects, such as the number of jobs created as a result of SBDC assistance to a business.) Audit objectives included assessment of output targets in the program s performance measurement system and the accuracy and reliability of program outcomes data. The following sections include findings and recommendations relating to these issues. Program Outputs and Performance Goals Measuring levels of activity within a program is a necessary component of a performance measurement system. Although output or activity measures cannot be used to assess economic impact for the SBDC program, they provide a means of tracking sub-center resource use and are important for analyzing other performance trends. The primary output measures used by the program include clients counseled, total counseling hours, number of training events, and training event attendance. As part of annual cooperative agreements with the SBA, targets are established for activity levels within each sub-center. These targets are used by state and federal program managers to analyze performance and identify areas for improvement. Analysis of SBDC Outputs Performance Targets Each sub-center is required to determine a reasonable performance target for clients counseled, total counseling hours, number of training events, and training attendance. These performance targets are reported on a quarterly basis. Sub-center performance targets were reviewed for program years 2001 through 2004 and compared with actual activity levels to determine if targets were established at a reasonable level. Analysis of trends in sub-center and network performance shows that some adjustments are necessary in the levels at which various targets are established. The following figure shows target and actual performance trends for the program between 2001 and Page 57

75 Chapter V Economic Trends & SBDC Service Areas Figure 10 Trends for Selected Output Performance Measures Calendar Years Source: Compiled by the Legislative Audit Division from department records. The program s use of performance targets has been improving, but some disparities still exist between growth trends for target and actual performance data. In relation to clients counseled, the target trend is moving ahead of actual performance, suggesting that performance expectations are not consistent with trends towards reducing client workloads. For counseling hours, the target and actual performance trends are converging, although there is still some degree of inconsistency between them. For both training events and attendance, target trends are increasing, but are not keeping-up with observed trends in actual performance as the program expands training opportunities. Page 58

76 Chapter V Economic Trends & SBDC Service Areas Adjustments in Performance Targets are Necessary Successful use of performance measurement benchmarking for management purposes depends on targets which are reasonably related to observed program conditions. Performance targets should be monitored to ensure they continue to reflect changes in program conditions. Where performance targets are either impossible to attain, or easily met or exceeded, they do not contribute to management s ability to effectively administer a program. SBDC program managers cannot make effective management decisions based on unrealistic performance targets. Depending on program management s expectations regarding future developments in these areas, adjustments should be made in performance targets to ensure they are both realistically achievable and sufficiently challenging. Recommendation #6 We recommend the department make periodic adjustments in SBDC sub-center annual performance targets to ensure they realistically reflect prevailing program conditions and management expectations regarding future trends. Verifying SBDC Outcomes and Economic Impact Quantifying the economic impact of SBDC services involves using reported client outcomes data to estimate the impact on the economy in terms of employment and investment growth in the private sector. To determine the accuracy of SBDC economic impact reporting, we reviewed program procedures for verifying reported client outcomes and estimating economic impacts through surveys and other studies. Program Approach to Outcomes Reporting is Conservative Overall, the SBDC program takes a conservative approach to reporting client outcomes. Observations from client files identified examples of potential client outcomes reported in client narratives, but not included as reportable outcomes in program reports. In general, these examples were instances where there was no evidence that actual employment/investment occurred, or the counselor-client contact was incidental and the effects of SBDC services were inconclusive. Even though this approach carries the risk of underreporting program outcomes, this is surpassed by the risk of falsely reporting the success of the program and overestimating its impact. Page 59

77 Chapter V Economic Trends & SBDC Service Areas Outcomes Data Cannot be Independently Verified Although the program takes a conservative approach to reporting client outcomes, significant problems remain in verifying the validity of this data. We could only identify one example during file review where documentation existed to verify reported client outcomes. Essentially, the program relies on good faith estimates provided by clients to determine the outcomes attributable to SBDC services. Reliance on self-reported data is common in economic development functions. In addition to the problem of verifying reported outcomes, another concern with this type of data is linking reported outcomes to program services. SBDC clients may report adding jobs as a result of receiving services, but this information is not necessarily based on consideration of other factors important in employment or investment growth. Existing Economic Impact Studies Rely on Self- Reported Data We reviewed two economic impact studies provided by the program. The first study was conducted by SBA and addresses economic impacts and other performance indicators for the SBDC program. The second study was conducted under contract for the Association of Small Business Development Centers (ASBDC). The ASBDC economic impact study relies on self-reported data from SBDC participants. While efforts are made to correct for response bias and other factors, the study suffers from an absence of verification of client data, low response rates, and reliance on attitudinal observation to link SBDC services and economic impacts. This does not suggest that existing economic impact studies have no value, or that the program should discontinue their use. The SBDC program should be credited with attempting these studies, which are too often lacking in other economic development functions. However, policy makers should exercise caution in their approach to outcomes and economic impact data provided using these methods. Conclusion: Reported outcomes and economic impacts rely on self-reported data and attitudinal observations. Page 60

78 Chapter V Economic Trends & SBDC Service Areas SBDC Program Outcomes Reporting The SBDC program, both in Montana and nationally, deserves credit for efforts made to institute a comprehensive performance measurement system to track outcomes and economic impact. Despite concerns over the reliability of reported outcomes and economic impact data, tracking data reported by clients is a useful tool for program managers. Program managers should track and report client outcomes, but the success of these efforts relies on consistency in the methodologies used to define and report outcomes. As discussed below, improvements are needed in SBDC program procedures to promote greater consistency in outcomes reporting. Reported Outcomes Data and Trends The program categorizes outcomes as either milestones or investments. Milestones quantify changes in business conditions for client companies and include sales and acquisitions, jobs created, jobs retained, and project size (an estimation of the overall capital value of a client s business project.) Investments quantify financial transactions by client businesses and include loans or financing secured from banks, government financing programs, owner equity, and other sources of capital. We identified four outcomes measures that have been consistently reported by the SBDC program over recent years; jobs created, jobs retained, project size and loan amounts. The following figure illustrates trends in these four outcomes measures reported by the SBDC program between 1998 and Page 61

79 Chapter V Economic Trends & SBDC Service Areas Figure 11 Trends in Reported Outcomes Data Calendar Years ,500 3,000 2,500 2,000 1,500 1, Employment Outcomes Jobs Created Jobs Retained Millions $ $400 $350 $300 $250 $200 $150 $100 $50 $0 Financial Outcomes Loans/Financing Project Size Source: Compiled by the Legislative Audit Division from department records. The most notable trend in reported outcomes data is the significant increase in all reportable outcome categories between 2003 and There is no clear explanation for why these increases have occurred. Significant fluctuations can sometimes be expected in outcomes data for a program like SBDC and could be related to projects or the inclusion of larger companies or projects resulting in Page 62

80 Chapter V Economic Trends & SBDC Service Areas temporary increases in outcomes levels. The sustained increase over recent years could also be related to changes in methodologies used by the program to define and report client outcomes. Reported Outcomes in Other States We also reviewed outcomes data for 15 other SBDC programs in states receiving the minimum federal funding appropriation. Data showed significant variations in performance levels between states. The following figure illustrates these variations using jobs created per client and loan amounts per client as examples. Figure 12 Selected Outcomes Data for Other Sources Federal Fiscal Years 2003 and 2004 Average Jobs Per Client Average Loan Per Client Source: Compiled by the Legislative Audit Division from department records. Both charts show average values based on the two years of available data. In terms of jobs created per client, Montana ranks near the top Page 63

81 Chapter V Economic Trends & SBDC Service Areas among peer states. Montana is the top-ranked state for combined loan dollars per client. Improvements in Performance Measurement System Needed The quality of the SBDC performance measurement system exceeds those used in many other economic development functions. Not all economic development programs collect the range of outcomes data reported by SBDC. However, review of reported data and data compilation and reporting standards for the program revealed inconsistencies in the program s methods for recording and reporting outcomes data. Concerns primarily relate to inconsistencies between sub-centers in use of the management information system to record and report client outcomes. Inconsistencies identified during audit work included the following: We observed at least three different methods for recording the number of jobs created by a client in the program s management information system. Distinctions between these three different definitions were not clear. Several sub-centers had not reported any outcomes data for the past three years of operations through the program s management information system. Other sub-centers had used the system consistently. In data reported to the lead center, it was not always possible to link reported outcomes to specific clients. Client descriptions were sometimes restricted to a general business or industry description. Page 64 Improvements in data definition and reporting parameters have already been started through the SBA and ASBDC, which has issued a compliance guide to all members. Discussions with department staff indicated the program is moving towards more consistent and effective use of the management information system to report outcomes data. The ability to link outcomes to specific client records and counseling narratives through the management information system provides a greater level of assurance that reported outcomes are an accurate reflection of the program s effectiveness. Continuing implementation of standardized outcomes data definitions and reporting parameters should ensure the quality and reliability of reported outcomes data is improved.

82 Chapter V Economic Trends & SBDC Service Areas Additionally, the lead center should take steps necessary to ensure SBDC counselors understand the new data definitions and reporting parameters and are implementing them correctly in reporting client outcomes. This should include quality control procedures to ensure reported outcomes are reasonably related to specific clients and counseling narratives. By continuing to monitor and analyze trends in reported outcomes, the lead center can begin to promote a more consistent approach to defining a reporting program outcomes. Recommendation #7 We recommend the department improve the consistency and reliability of reported SBDC outcomes data by: A. Ensuring sub-center counselors are familiar with standardized data definitions and reporting parameters developed by the Association of Small Business Development Centers; and B. Developing quality control mechanisms to verify compliance with outcomes data reporting procedures. Overall Conclusion The comparison group study discussed in chapter II of this report provides the most reliable method for assessing the impact of SBDC services on participating businesses. However, program managers do not always have the ability to access the relevant data or resources to undertake complex analysis of outcomes. As an alternative, measuring performance using output and outcomes data can help managers understand and improve program operations. All forms of performance measurement are useful management tools, even where they rely on self-reported data. However, management needs to commit to using these tools consistently before their value is realized. By improving the consistency of its outputs and outcomes performance measurement systems, the SBDC program can ensure it has continuing access to quality operational data. Page 65

83 Page 66

84 Appendix A - Audit Approach Audit Scope To establish audit scope, we initially relied on the guidance provided in the performance audit priority narrative reviewed by the Legislative Audit Committee for the 2005 biennium. The audit priority narrative addressed assessment of SBDC program outcomes and the availability of similar services in the private sector. We also reviewed federal statutes and other materials from the Small Business Administration (SBA) and interviewed officials in the Department of Commerce to identify additional issue areas. Audit scope was limited to the extent that federal oversight of the SBDC program provides assurance of appropriate management control in several areas. The SBDC program and the activities of sub-centers in the Montana network are subject to regular financial and programmatic reviews conducted by the SBA and the Association of Small Business Development Centers (ASBDC). For this reason, we excluded from audit scope issues relating to internal management controls in the lead and sub-centers, financial management, and compliance with contract requirements relating to host agency cash and in-kind contributions. Audit Methodologies In response to audit objectives, we developed methodologies outlined in the following sections. Audit Planning We obtained and reviewed federal statutes and regulations authorizing the SBDC program, and Montana statutes relating to the SBDC program and business assistance/economic development in general. We reviewed general fiscal/budget information from Legislative Fiscal Division and other sources showing program budgets, revenues, expenditures, and federal funding allocations and state/local matching funds requirements for SBDCs. Interviews were conducted with key program staff to discuss general program organization and administration, management controls, management information systems, and other basic program information. We also contacted federal government officials with oversight responsibility for the Montana SBDC program to discuss federal oversight procedures and other relevant issues relating to administration of the program. Audits, research reports and other information from the Page A-1

85 Appendix A - Audit Approach national ASBDC, state and legislative audit offices, the National Conference of State Legislatures, and other public policy organizations was reviewed to identify issues relevant to the SBDC program. SBDC Service Provision The comparison group study of program economic impact analyzed data for businesses receiving SBDC counseling services between January 1, 1999 and December 31, Participating businesses all received at least three hours of program counseling. The following additional restrictions were applied in selecting businesses for the study; a valid description of the type of business needed to be included in program data; all businesses were required to be active as of August 2005; and only businesses formally organized as corporations, sub-s corporations or limited liability companies were included. Each SBDC beneficiary business was matched with nonbeneficiary business based on industry sector, geographic location (urban or rural), and number of years in operation. Data for nonbeneficiary businesses was sourced from a commercial database of Montana business entities. All matching businesses were required to meet the same selection requirements discussed above, and these elements were verified using the Montana Secretary of State s business entity records database. Economic impact data was obtained from the Department of Revenue s corporation income tax and the Department of Labor and Industry s unemployment insurance databases as available for the period 1999 through The comparison group study was not designed to control for every possible variable affecting private sector business performance. Certain unobservable variables may be important in understanding the performance of businesses included in comparison groups. Methodologies provide a reasonable level of assurance that selection bias was limited to every possible extent. Analysis and results are applicable to services provided by the SBDC program between January, 1999 and December, Page A-2

86 Appendix A - Audit Approach Alternative Service Providers To define the scope of SBDC counseling activities, we interviewed department staff and program counselors regarding the types of clients and typical counseling activities. We also reviewed individual files for 62 clients receiving services in state fiscal year 2004 in five different sub-centers. Data from the program s management information system was also reviewed to identify trends in the type of counseling services provided between calendar years 1998 and Provision of similar services in the private sector was analyzed using data and other background information compiled by United States Census Bureau as part of the 2002 Economic Census. Estimated costs for the provision of private sector services was based on a national survey of management consulting firms conducted by an independent research firm in Analysis of public sector provision of business technical assistance was based on program and other background data provided by 10 different programs or organizations. Public sector service providers were identified through discussions with department staff, and SBDC counselors and other host agency employees. Administrative Compliance and Oversight We obtained and reviewed federal laws and regulations relating to administration of the SBDC program and SBA oversight requirements. To determine program compliance status, we reviewed internal policies and procedures, SBA cooperative agreements, staff position descriptions, on-site inspection reports for sub-centers, and host agency contracts. Regional SBA staff provided financial audits and programmatic reviews conducted for the Montana SBDC program in program year Assessment of compliance with SBA business size regulations included review of client records stored in the program s management information system. We identified and reviewed records for 16 clients with more than 100 employees. Information from various external sources was used to identify the total number of employees and/or the annual receipts for each client business. Information on the structures/organization and administration of SBDC programs in other states was sourced from the SBA, ASBDC or directly from lead centers in other states. Data on the Page A-3

87 Appendix A - Audit Approach structure/organization of Montana s SDC network was compared with nine other western or southwestern states. Analysis of federal and state funding compared Montana data with information for 11 other states receiving at or near the minimal federal funding allocation in federal fiscal year SBDC Counseling and Training Activities SBDC counseling and training activity data was sourced from the program s management information system. Analysis was based on records from database tables downloaded prior to a major system conversion in early These tables contained records for all clients and training activities occurring since the program began using the current information system. We performed a review of 62 individual client files and 26 training event files in five different subcenters to validate the data sourced from the information system. Analysis of client activity and demographics addressed trends over time and disaggregated data for individual sub-centers or sub-sectors of the whole population. Analysis of trends in veteran participation levels also relied on data from the United States Census Bureau for veteran populations in Montana counties from the 2000 census. Significant trends in client counseling and training activities were discussed with department staff and SBDC counselors during interviews. Economic Trends in Program Service Areas Comparison of economic trends in SBDC service areas with program activity levels relied on the United States Census Bureau s County Business Patterns (CBP) and Non-Employer Statistics (NES) data series. Annual data for the number of business establishments was collected for all the industry sectors represented in the CBP and NES series for 1997 through 2002 (values for calendar years 2003 and 2004 were projected based on prevailing trends.) County-level data was aggregated by SBDC service area and aggregated to show trends over the period of study 1998 through Business establishment data for SBDC service areas in each year was used to calculate a projected level of program participation based on the service area s share of all establishments and the total number of statewide clients served by the program. The variance value between the projected Page A-4

88 Appendix A - Audit Approach and actual level of client participation expresses the extent to which each sub-center performed above or below an ideal level of activity. Program Outputs and Outcomes Reporting Assessment of program outcomes and economic impact involved analysis of data, reports and other information released by the SBDC program, review of performance measurement data compiled by the SBA, and discussions with department staff and SBDC counselors. We reviewed program data showing quarterly and annual program output performance targets submitted as part of cooperative agreements with the SBA. Sub-center performance target trends (median annual growth between 2001 and 2004), were compared with actual performance trends to determine if performance targets reasonably reflected prevailing program conditions. Analysis of program outcomes data relied on statistics previously published by the SBDC program (outcomes data recorded in the program s management information system was not suitable for trend analysis due to omissions and other inconsistencies reported by sub-centers.) We also reviewed information supplied by the ASBDC in relation to outcomes data definitions and reporting parameters, and SBA data for reported outcomes in other states in federal fiscal years 2003 and Page A-5

89 Department Response Page B-1

90 Page B-2

91

92

93

94

95