Meeting the Challenges of Designing the Kauffman Firm Survey: Sampling Frame, Definitions, Questionnaire Development, and Respondent Burden

Transcription

1 Meeting the Challenges of Designing the Kauffman Firm Survey: Sampling Frame, Definitions, Questionnaire Development, and Respondent Burden Janice Ballou 1, David DesRoches 1, Zhanyun Zhao 1, and Frank Potter 1 Mathematica Policy Research, Inc. 1 Abstract The challenges in designing the Kauffman Firm Survey (KFS) provided an opportunity to study and evaluate the methods for conducting establishment surveys. This paper highlights some of the noteworthy processes used during the design phase of the project, including setting up an advisory committee, conducting the literature review, developing the sample frame and design, developing criteria to define a new business, and designing the questionnaire. Cognitive interviews were conducted, and pilot testing included experimenting with respondent incentives (pre- versus postpayments) and data collection mode (self-administered via the web versus telephone interviews). The KFS design process reinforced the value of spending time and resources to address key issues prior to the full-study implementation that required screening 32,500 sampled businesses to identify 4,930 eligible new businesses. Keywords: Establishment survey, Survey design process, Sampling frame, Questionnaire development 1. Introduction Entrepreneurship plays a major role in the economic growth of the country, and accurate information about the development and sustainability of new businesses is essential to developing public and private programs that encourage new business development. However, achieving success in obtaining this information is challenging. Surveys of new businesses tend to be difficult to implement and have typically produced low response rates because of the difficulty of obtaining the cooperation of new business owners. Surveys of new businesses have also faced the complexity of defining what constitutes a new business and when a new business begins operations, events that lend themselves to subjectivity if not carefully defined for business owners. Further, few previous business surveys collected information about the dynamics of business development, since longitudinal surveys of new businesses faced the issue of business attrition. As part of its mission to advance entrepreneurship and the study of new business creation and development, the Ewing Marion Kauffman Foundation (the Foundation) sponsored the Kauffman Firm Survey (KFS) to address these informational and methodological gaps. Because of this commitment, we were able to execute a rigorous process to develop the KFS. Methods papers often focus on end results, and the process used to get to these results is not given much attention. Since the KFS provided the opportunity to execute an almost ideal sequence of actions to develop both quality measures and useful substantive information, this paper can be viewed as a formative evaluation that can guide others who are designing establishment surveys. 2. From the Beginning: Starting Point for Survey Design and Development Designing a survey of new businesses is challenging because of the methodological issues related to finding and identifying eligible businesses and developing questionnaire items that accurately measure the key concepts related to business development. The starting point for the KFS was based on prior establishment surveys. The sample frame was a Dun & Bradstreet (D&B) database. The eligibility criterion to be considered a new business in a specific reference year was being listed for the first time in the D&B database. We also assumed that the survey respondent would be a founder or owner of the business who was also involved in business operations, and that the questionnaire items would be similar to those in the 1992 Economic Census. However, as the KFS survey design process evolved, multiple changes were made in the original research design proposal. From this starting point, a process began that included a thorough review of new business sampling frame issues, definitions and criteria for what constitutes a new business, identification of priorities for questionnaire items, and a review of data collection techniques to maximize response rates. A comprehensive and collaborative process was used before the KFS was finalized. 464

2 3. Are We on the Right Track? Document Review and Technical Advisory Group Two steps were employed to inform the design process and to test the validity of the assumptions in the proposed research: (1) a review of business and relevant literature and (2) consultation with an advisory group composed of probable users of the KFS data Literature Review The literature review included about 60 articles and related surveys that focused on business statistics and the dynamics of business formation. In particular, we reviewed the survey instruments used for the 1992 Economic Census, the 1998 Survey of Small Business Finance, and the Panel Study of Entrepreneurial Dynamics. 3.2 Technical Advisors In total, more than 20 technical advisors contributed to the development of the KFS. These advisors were selected because of their interest, expertise, and scholarship related to entrepreneurship. In addition, it was expected that the core users of the KFS data files would be among this group, so they were given an opportunity to inform the process. Also, the advisory group outlined a wish list of information that would best meet the needs of academic researchers, government agencies, and public policy decision makers who would use the KFS data. 1 To begin the KFS development and design process, an initial meeting in May 2004 was held with a core advisory group. Based on this meeting and subsequent meetings, the following concepts were agreed upon to frame the development of the KFS: (1) the data collected would be relevant to a pure cohort of businesses that started in a single targeted year, (2) the business not any individual owner or founder would be the focus of the information collected, (3) financial information related to business formation would be the main analytic objective, and (4) a longitudinal survey design was needed to inform an understanding of business development dynamics and sustainability. To facilitate the ongoing process, the advisory group was represented by a single principal investigator in the core development group along with Mathematica Policy Research, Inc. (MPR) and the Foundation project team. At various phases of the survey development, however, other participants were consulted 1 During the same time period as the KFS development, the Foundation was also funding a major effort by the National Academies of Science (NAS) to identify the informational needs related to business surveys and other information needed to study new business development. Several NAS participants were also KFS advisors. to assist in crafting the final survey instrument and responding to methodological questions that surfaced. 4. Who Should We Interview? Sample Design Issues Prior to the work on the content areas of the KFS questionnaire, there was a need to understand how well the definitions developed by the core advisory group matched up with the sample frame from the D&B database. Discussions among the Foundation, MPR, the principal investigator, and the advisory group about criteria to identify eligible businesses resulted in the decision to conduct a pilot test. The pilot test was needed because little was known about the expected eligibility incidence. To begin the baseline survey without knowledge of the eligibility incidence and the costs related to higher or lower incidence would be risky. 1 was conducted to determine the likely eligibility of businesses in the D&B sample and to learn more about sample reliability, including business attrition and business owner gender identification A 1 assessed multiple criteria being considered for eligibility. 1A was conducted in August 2004 and was used to test the incidence of state unemployment insurance (UI) and Federal Insurance Contributions Act (FICA) tax payments among the D&B sample. A small sample of businesses (720) was selected from the 2003 D&B file in July Interviews were completed with 173 of the selected businesses to determine UI and FICA payments, and if these were made for the first time in The main finding from 1A was that 20 percent of businesses reported making either UI or FICA payments during the reference year of 2003, which was far lower than expected. However, the reliability of D&B identification of an active business was 93 percent and owner gender was 85 percent for women and 91 percent for men B Because of this low incidence, the eligibility criteria were expanded. In addition, the project team was concerned about using UI and FICA payments exclusively, since these measures are driven by having employees and would result in underrepresentation of nonemployer, single-owner businesses. The information from the business survey literature and from members of the advisory group indicated that a sizeable minority of new businesses have no employees, and not capturing enough of such businesses would increase bias in the survey estimates. To test additional eligibility criteria at a reasonable cost and within a short time frame, 465

3 1B recontacted the businesses that had previously been interviewed and asked additional questions about legal business status (sole proprietorship, general partnership, limited partnership, C-corporation, subchapter S- corporation, limited liability company), acquisition of an Employer Identification Number (EIN), and use of an Internal Revenue Service Schedule C or C-EZ as part of the owner s 2003 income tax return. The eligibility criteria tested in s 1A and 1B increased incidence to 36 percent and were incorporated into the KFS questionnaire (Table 1). Table 1: Eligibility Criteria 1A 1B 5. What Questions to Ask? Questionnaire Development Process 2 Final Legal Status X X X EIN X X X IRS Schedule C or C-EZ X X X State UI Payments X X X FICA Payments X X X A comprehensive and iterative process was used to develop the final questionnaire. The initial draft KFS questionnaire was crafted using the matrix of topics suggested by the advisors and relevant questionnaire items from prior studies. An explicit goal suggested by the advisors during the design process was harmonization of the KFS with other business surveys. Six in-person cognitive interviews were conducted with eligible new business owners to assist in evaluating the survey instrument. Cognitive interviews were conducted to identify difficulties in question comprehension, perceptions of the response task, memory recall strategies, difficulties in selecting a response choice, and interpretations of question reference periods. In addition, they provided information to understand better the respondent burden involved with completing the questionnaire and how business owners would respond to potentially sensitive business financial questions. Also, key concepts used in the questionnaire were reviewed to make sure they were understood. For example, the cognitive interviews assessed if business owners could report accurately when they were asked to retrieve information for the reference periods in calendar year 2003 or on December 31, The cognitive interviews provided a number of suggestions about how business terms included in the questionnaire could be better defined for the new business population. These suggestions were incorporated into the questionnaire prior to Are We Ready to Go? Comprehensive Pretesting The information from 1 and the cognitive interviews, as well as the desire to test the questionnaire fully, resulted in the recommendation for a comprehensive pretest ( 2) prior to conducting the KFS Baseline Survey. In addition to continuing to learn about business eligibility based on the established criteria, 2 was used to (1) test the length of questionnaire; (2) review response distributions, missing and inappropriately skipped questions, and incomplete questionnaires; and (3) conduct several methodological experiments Overview 2 was conducted from January 2005 to March 2005 among a sample of 6,235 businesses selected from the 2003 D&B frame of business starts. We completed 400 interviews during 2, and a total of 745 businesses were screened out as ineligible. 2 produced a lower level of response than expected and informed decisions about methods to use to increase participation and to lower the cost to complete interviews Experiments 2 also provided an opportunity to conduct experiments on methodological approaches that could be used to improve data quality. The survey research literature suggests that incentives can increase participation. To inform the incentive decision, we conducted an experiment during 2 with four types of incentives: (1) $1 prepaid/$50 postpaid, (2) $1 prepaid/$0 postpaid, (3) $0 prepaid/$50 postpaid, and (4) $0 prepaid/$0 postpaid (Table 2). Each group was analyzed to identify the most cost-effective incentive approach for the full KFS. Postpaid Table 2: Incentive Experiment Prepaid $0.00 $1.00 Nothing 20% 22% $ % 30% Total 100% 466

4 Another 2 experiment was the use of two different methods of data collection a self-administered web questionnaire and a telephone interview. We used a forced-web approach in which the advance materials mentioned only the web survey, since having sample members respond using the web questionnaire is the most cost-effective method to obtain information. Using this method assisted in estimating the expected number of completed interviews using the web questionnaire, which was useful in preparing for the Baseline Survey. It also provided information to compare the two methods of data collection and assess possible mode effects on data quality Results Several decisions were made based on the results of 2. The length of the questionnaire was reduced from a 50-minute average (which was found to be a substantial time commitment from new business owners) to a 20- minute average time to complete by telephone. Shortening the questionnaire accomplished two objectives: (1) focusing on the financial items that were the main analytic priority and (2) reducing the length of time to complete the questionnaire to encourage participation and increase the response rate. The advisors were included in making decisions about which questionnaire items to omit. Using the results of 2, we conducted important discussions about the analytic contribution of each item to the field of entrepreneurship studies. The process was thorough and included a Delphitype approach 2 to document advisor questionnaire item recommendations. The project team carefully assessed the advisors' suggestions and responded with complete documentation on the rationale for KFS inclusion or exclusion. Information about 2 results was shared with the advisors to keep them informed of the status of the questionnaire items and decisions that had to be made prior to agreement on the final KFS Baseline Survey instrument. Based on the incentive experiment, we decided to use only a $50 postinterview payment and no prepaid incentive. No statistical difference existed between the prepayment incentive groups in terms of completion. There was also no statistical difference in completions among the postpayment groups, although evidence of a directional effect does exist (8 percent completed the interview among the incentive treatment group versus 5 percent in the control group). Given the expected difficulty of obtaining a high response rate during the 2 The Delphi approach seeks consensus by solicitation and comparison of the views of experts. In the KFS, the questionnaire was provided to members of the technical advisory group, and the comments from each member were evaluated independently before grouping comments on particular questions. Baseline Survey, it was decided that offering the incentive would be more cost effective than additional followup to nonresponding businesses. Based on the cognitive interviews, comments provided by the advisors, and the data and methodological lessons learned from 2, the content of the KFS Baseline Survey questionnaire was finalized in May The first two sections of the questionnaire were devoted to screening for an appropriate respondent and to ensuring that the business was eligible for the survey. The remaining sections covered business characteristics, strategy and innovation, business organization and employee benefits, business finance, and work behaviors and demographics of the owners. 7. What We Learned: Observations on Meeting the Challenges The research, collaboration, and analysis conducted prior to launching the KFS were very useful. Even though no survey will meet all analytic needs, users of the KFS data will be assured that it was developed using a comprehensive and inclusive process and data from the testing were used to make thoughtful, rather than arbitrary, decisions. Academic researchers and policymakers may use different criteria for what constitutes a new business. Although there might be preferred alternative definitions, users can be sure that the KFS data include businesses that have met a specified set of criteria and will know exactly what the data represent. Using a D&B sampling frame was a practical solution to finding new businesses. s 1 and 2 provided information to identify the strengths and weaknesses of D&B information, so that the trade-offs discussed in using this sample frame could be thoroughly investigated. Overall, following this process resulted in the ability to anticipate possible costly problems, such as not knowing the expected incidence of various eligibility criteria. Having a well-crafted questionnaire resulted in a higher response rate than is experienced by most establishment surveys, and higher quality data because of lower item nonresponse, in particular, reduced missing data on key financial questions. Acknowledgments The authors would like to acknowledge the support of the Ewing Marion Kauffman Foundation and the project team that worked with us to develop the KFS: Robert Litan, vice president of research and policy; Robert Strom, director of research and policy; and E.J. Reedy and Alyse Freilich, project directors. 467

5 References Haltiwanger, John, Lisa M. Lynch, and Christopher Mackie (eds.). Understanding Business Dynamics: An Integrated Data System for America s Future. Washingon, D.C.: The National Academies Press,