1 G Program Development and Evaluation Questionnaire Design: Asking questions with a purpose Ellen Taylor-Powell Program Development and Evaluation Specialist May 1998
2 Originally published with Mary G. Marshall Texas Agricultural Extension Service The Texas A&M University System College Station, Texas
3 TABLE OF CONTENTS Constructing a questionnaire....2 Kinds of information...2 KNOWLEDGEÑWhat people know...2 BELIEFSÑATTITUDESÑOPINIONS BEHAVIORÑWhat people do...3 ATTRIBUTESÑWhat people are...3 Wording the questions Types of questions...5 Close-ended questions with one-choice answers Formatting the questionnaire Pretesting the questionnaire References...15 Appendix...17
4 2 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Constructing a questionnaire A questionnaire provides a tool for eliciting information which you can tabulate and discuss. In many evaluations, a questionnaire serves as the major source of information. Writing questions and constructing a questionnaire takes time and attention. Before you begin, it is essential to know what kind of evidence you need to fulfill the purpose of the study and to know how the information will be used. Make a list of what you want to know. What do you really want to find out? What do you want to achieve with the questionnaire? Check to see if the information you need is already available somewhere else. DonÕt ask a question unless it has a use; that is, unless it relates to the purpose of the study. Collecting too much information adds to your time and expenses and can produce an information overload. Eliminate all the Ònice to knowó items that arenõt really essential. Eliminate ambiguous questions as well. From the beginning, think through what you will do with each piece of information. What do you want to be able to say? Do you expect to use frequencies, percentages, rankings, multivariate analysis, narrative remarks? As you write questions, try to view them through your respondentsõ eyes. Will the question be seen as reasonable? Will it infringe on the respondentõs privacy? Will the respondent be able and willing to answer the question? Be selective and realistic. Know what information is needed, why, and how you plan to use it. Kinds of information A questionnaire can help you obtain information about what people do, what they have, what they think, know, feel, or want. Four different types of information may be distinguished. Any one or a combination of these types may be included in a questionnaire. KNOWLEDGE What people know; how well they understand something This type of question asks about what people know. Knowledge questions offer choices such as correct vs. incorrect or accurate vs. inaccurate. They may ask what respondents believe is true or factual, or about awareness. For example: What is the major cause of accidental deaths among children inside the home? The most effective weight loss plan includes exercise. The ideal refrigerator temperature is. BELIEFS ATTITUDES OPINIONS These terms refer to psychological statesñthe perceptions people hold, their thoughts, feelings, ideas, judgements, or ways of thinking. Questions may elicit respondentsõ perceptions of past, present or future reality, their feelings about a subject, or their opinions. For example: Do you favor or oppose the reclassification of forestry land? In your opinion, does positive self-esteem prevent drug abuse among adolescents? Do you think that lower prices would increase beef consumption? What do you consider the biggest challenge facing our community in the next five years?
5 Q U E S T I O N N A I R E D E S I G N 3 BEHAVIOR What people do Questions on behavior ask people what they have done in the past, do now, or plan to do in the future. For example: Have you ever attended an Extension program about nutrient crediting? Do you scout fields for pest problems? How are you currently using the information gained in the Healthy Eating workshop? ATTRIBUTES What people are; what people have Attributes are a personõs personal or demographic characteristicsñage, education, occupation, or income. Attribute questions ask people about who they are, rather than what they do. For example: Where do you currently live? How many children do you have? What percentage of your household income comes from off-farm employment? To write meaningful questions, be clear about the objectives and type of information desiredñwhether it is information about knowledge, attitudes/beliefs/opinions, behavior, or attributes. Otherwise, the questionnaire may elicit opinions when the actual intent is to document behavior. Likewise, questions related to each type of information present different writing problems. Questions concerning attitudes tend to be more difficult to write given the complexity underlying most attitudes. Pay careful attention to wording. In contrast, questions about knowledge, behaviors and attributes are more straightforward. Remember, the response or information you obtain is only as good as the question. If you donõt get the type of information you want, it is probably because you didnõt ask the right question! Wording the questions Wording the questions to obtain the intended information and to be understood by all respondents is a challenging task. When you write questions, consider three things: 1) the particular people for whom the questionnaire is being designed; 2) the particular purpose of the questionnaire; and 3) how questions will be placed in relation to each other in the questionnaire. Some suggestions appear below. (adapted from Sawer, 1984; Dillman, 1978; Newsome, n.d.) Use simple wording. Adapt wording to the vocabulary and reading skills of your respondents but donõt talk down to them. Are any words confusing? Do any words have double meanings? Avoid the use of abbreviations, jargon, or foreign phrases. We use a lot of jargon in Extension. Will the respondents understand terms such as ÒCNRED,Ó Òlearning experiences,ó Òlife skills,ó or Òfocus groupsó? Be specific. A question about older youth should specify what age or grade is considered Òolder.Ó Likewise, in the question ÒHow many times did your 4-H club meet last year?ó specify whether Òlast yearó refers to 1994, 1994Ð1995, the last 12 months, or September 1994ÐAugust Use clear wording. Words such as ÒregularlyÓ and ÒoccasionallyÓ mean different things to different people. Some vague terms include: majority (more than half of what?); often (daily? twice weekly? weekly?); government (state? federal? local?); older people (how old?).
6 4 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Include all necessary information. In some cases, respondents may not know enough to adequately answer a question. For example: ÒDo you agree or disagree with the proposed plan to expand the role of 4-H volunteers in our community?ó Respondents may not know what the plan is. Provide a statement summarizing the plan. Avoid questions that may be too precise. PeopleÕs lives are usually not so orderly that they can recall exactly how many times they ate out last year or how many Extension meetings they attended in To help respondents formulate an answer, the response category might provide a range to select from, for example, 0Ð5, 6Ð10, 11Ð15, etc. Phrase personal or potentially incriminating questions in less objectionable ways. Being asked to indicate drug use, income level, or ethnic background may be objectionable to some respondents. One method is to ask respondents to select from among broad categories (income less than $10,000, $10,000Ð$20,000, $20,000 and over, etc.) rather than specifying precise information. A series of questions may also be used to soften or overcome the objectionable nature of certain information. Avoid questions that are too demanding and time consuming. Examples of such questions are, ÒPlease rank the following 15 items in order of their importance to youó or ÒIn 25 words or less, what is your philosophy of 4-H?Ó Use mutually exclusive categories. Make sure that only one answer is possible. In the example: ÒHow did you hear about the Extension seminar?ó the response categories are: Òfrom a friend, from a relative, from the newspaper, at work, from the county office, at an Extension meeting.ó The respondent may have heard about the Extension seminar from a friend at work, for example, so that more than one answer is possible. Avoid making assumptions. Questions such as ÒHow many children do you have?ó or ÒDo you prepare beef when you invite friends over to eat?ó make assumptions about the respondentsñthat they have children and invite friends over to eat. A better set of questions would start with the first question establishing the situation, followed by the question of interest. For example: ÒDo you have children?ó ÒHow many children do you have?ó Avoid bias in questions. Biased questions influence people to respond in a way that does not accurately reflect their positions. A question can be biased in several ways: (1) when it implies that the respondent should be engaged in a particular behavior; (2) when the response categories are unequal or loaded in one direction; (3) when words with strong positive or negative emotional appeal are used, such as Òfreedom,Ó Òequality,Ó Òboss,Ó Òbureaucratic,Óetc. Here are some examples of biased questions: 1. More farmers in Saymore County are using Superb than any other variety of alfalfa. Do you use Superb? 1. No 2. Yes This question implies the respondent should be using Superb. 2. How would you rate the housing in which you live? 1. Satisfactory 2. Good 3. Excellent No negative options provided. 3. Do you agree that funding for Extension in your county should be increased? 1. No 2. Yes
7 Q U E S T I O N N A I R E D E S I G N 5 This is a leading question. A better question would state: Do you agree or disagree that Extension funding should be increased? (Circle one.) 1. Strongly disagree 2. Disagree 3. Agree 4. Strongly agree Avoid double-barreled questions. ÒDid the poultry production seminar help you to identify ways to improve the sanitation and increase the nutrition of your cage bird operation?ó ItÕs better to ask about ÒsanitationÓ and ÒnutritionÓ separately. Other questions may be too ambiguous; for example: ÒDo you favor legalization of marijuana for use in private homes but not in public places?ó This gives no opportunity for people to respond in favor of both places, to oppose both places, or to oppose home but favor public use. Make the response categories clear and logical. Too often the answers are confusing, not in logical order or spaced so that numbers or figures are hard to interpret. For example: Poor spacing 1.0 acres acres acres acres acres 6.1,000 acres Poor logic 1. 1,000 acres acres acres acres acres 6. 0 acres Better 1. 0 acres acres acres acres acres acres acres and over Use complete sentences. Trying to keep questions simple and concise may result in questions that are cryptic and easily misunderstood. Plan ahead. Identify each question and each response item with a number or letter for easy tabulation. Types of questions Questions can be open- or close-ended. The following is adapted from Sawer, Open-ended questions allow respondents to provide their own answers. This gives them the opportunity to express their own thoughts, but also requires more effort in terms of their responses. Open-ended questions tend to produce varieties of answers and are more difficult to analyze. Close-ended questions list answers, and respondents select either one or multiple responses. These questions produce more uniform answers than open-ended questions, but depend upon your knowing and including all relevant responses in the list. Responses to close-ended questions must be exhaustive and also mutually exclusive in providing for the selection of a single response. Examples of open- and close-ended questions are explained further on the next several pages.
8 6 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N A question on one topic structured in different ways 1. Open-ended What would you like to see as the main program emphasis next year? 2. Close-ended with ordered responses How important to you are each of the following possible program emphases? (Circle one for each item.) None Little Some Much A. Effective parenting B. Child development C. Guidance & discipline D. Communications Close-ended with unordered response choices Which of these four topics would you most like to see as the primary program emphasis next year? (Circle number of your answer.) 1 Effective parenting 2 Child development 3 Guidance and discipline 4 Communication 4. Partially close-ended What topic do you feel should be the main program emphasis for next year? (Circle number of your answer.) 1 Effective parenting 2 Child development 3 Guidance and discipline 4 Communication 5 Other (please specify) (Dillman, 1978) Open-ended questions An open-ended question is often the easiest way to ask for information, but the responses are not easy to analyze. Answers are likely to be varied so you will need to categorize and summarize them. Think about how you will analyze the narrative data. Open-ended responses can be used to: Stimulate free thought, solicit creative suggestions, or recall information learned. Probe for more detail. Examples: 1. What do you think should be done to improve the 4-H program in this county? 2. Name the five basic food groups. 3. Please indicate how you intend to use the information obtained during the workshop. 4. We are interested in any other comments you might have concerning your role as a volunteer leader. Please write in the space below any thoughts youõd like to share with us. When asking for a numeric response, include the unit of measurement to be used. 5. Please list the number of acres (if any) of temporary pasture you planted in A. Acres of wheat B. Acres of oats C. Acres of rye grass D. Acres of clover E. Acres of summer annuals F. Other, please specify Open-ended questions are also appropriate when respondents are asked to supply a specific answer and a large number of responses are possible (see example 3 above) or when all the possible answers are not known. They are often used at the end of a questionnaire to ask respondents for additional comments (example 4 above).
9 Q U E S T I O N N A I R E D E S I G N 7 Close-ended questions There are a variety of ways to write closeended questions. Some require answers that fall along an implied continuum (as in rating scales); others supply answers in no particular order (lists). Some questions employ multiple choice options (Òcheck all that applyó); others provide relevant answers but allow respondents to add others not in the list. The following section gives examples of close-ended questions. Two-option responses This is the simplest response format. Options may include: NoÐYes, DisagreeÐAgree, FalseÐTrue, OpposeÐFavor. Example: 1. Do you remove the clippings from your lawn after mowing? 1 No 2 Yes Depending upon the information you desire, this may be the most appropriate format. It is often used as the first question in a series of questions on one topic. However, using a rating scale or a ranking (when appropriate) elicits more information. Starting with either positive or negative response options appears to have little effect on response. Neither does it matter whether Yes or No is listed first. But you do need to be consistent in the order you follow throughout the questionnaire. One best answer These questions can be used to solicit information or to test knowledge. They are appropriate when all relevant choices are known and listed. Respondents are provided with a list of answers and asked to check or circle the choice they feel is the best. Responses are independent of each other, rather than gradations along a continuum. Example: 1. What does the word ÒnutritionÓ mean to you? (Circle one number.) 1 Getting enough vitamins 2 The food you eat and how your body uses it 3 Having to eat foods I don t like 4 Having good health Rating scale Often, respondents are asked to indicate their choice at the most appropriate point on a scale. Whether you use a scale of three, four, five or more categories depends on the question, the amount of differentiation that is possible and desirable, and the respondentsõ capacity to answer. Examples: 1. To what extent do you agree or disagree with the new zoning code? (Circle one.) 1 Strongly disagree 2 Mildly disagree 3 Neither agree or disagree 4 Mildly agree 5 Strongly agree 2. When purchasing new herd bulls, how important are the following traits in your selection process? (Circle one number for each selection trait.) Selection Of little Highly trait importance important A. Performance B. Conformation C. Pedigree D. Breed For greater differentiation, use a numerical scale from 0 or 1 to some number (see the second example above). A five-point option series seems to be best for measuring attitudes; a four-point option series appears useful for ratings (excellent, good, fair, poor). Some people may relate best to a 10-point scale.
10 8 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Another decision youõll need to make is whether to use an even or odd number of response options. An odd number of categories provides a middle or neutral position for the respondent to select, while an even number forces the respondent to take sides. This is appropriate when you want to know in what direction the people in the middle are leaning. Below are some examples of rating response categories. Very dissatisfied Somewhat dissatisfied Somewhat satisfied Very satisfied Strongly unfavorable Generally unfavorable Uncertain Generally favorable Strongly favorable Decreased Stayed the same Increased No help at all Slightly helpful Fairly helpful Very helpful Strongly disagree Disagree somewhat Uncertain Agree somewhat Strongly agree Poor Fair Good Excellent Five categories are about the most you should use when listing words in the responses. Remember to keep the positive and negative options balanced. Ordered choice In this type of question, the responses are usually intended to measure degree or intensity in an ordered sequence or scale. Ordered choice questions are particularly suited for evaluating attitudes. They are appropriate when the topic is well-defined and the choice represents a gradation along a single dimension. Examples: 1. Within your 4-H club, describe the extent to which you were included in making important decisions. (Circle one number.) 1 Never 2 Rarely 3 Sometimes 4 Often 2. How do you feel about this statement, ÒI wish this community had more outdoor recreation centers?ó (Circle one number.) 1 Strongly disagree 2 Mildly disagree 3 Neither agree nor disagree 4 Mildly agree 5 Strongly agree Responses must reflect a clear difference and be balanced both positively and negatively. A poor set of responses would: Òdecreased, stayed the same, increased a little, increased somewhat, increased a lotó (only one negative and three positive choices are given). It is not necessary to use the same response categories for each question, but do it where possible. Most importantly, choose responses that are appropriate to the question. Other, please specify Here the respondent is offered a choice of an answer plus the opportunity to enter his/her own answer under Òother, please specify.ó This protects you against leaving out an important answer choice. It also means that you will have narrative text to analyze. Think about what you will do with these responses. Too often this type of information is never used because it cannot be added up easily.
11 Q U E S T I O N N A I R E D E S I G N 9 Examples: 1. What do you consider the main responsibility of your county 4-H agent? (Circle one number.) 1 Work with people who request help 2 Work with 4-H members 3 Work with volunteer 4-H leaders 4 Plan and organize county youth events 5 Organize and expand new 4-H clubs 6 Other, please specify 2. Which of these community recreational facilities do you most frequently use? (Circle one number.) 1 Parks 2 Tennis courts 3 Swimming pools 4 Other Items in a series When several questions use the same response category, it is possible to present the responses in a table, rather than write separate questions for each. Example: 1. How often do you eat the following meats? (Circle one number for each meat.) Once/ 1 3 times 4 6 times Never week week week Daily A. Beef B. Lamb C. Pork D. Poultry E. Fish Paired comparisons Respondents are asked to compare one item to another, usually expressed in terms of Òeither/orÓ or one item ÒversusÓ another. Example: 1. In comparing beef to other meats, which does your family use more often? (Choose one from each comparison and circle the number.) 1 Beef OR 2 Poultry 3 Beef OR 4 Lamb 5 Beef OR 6 Pork 7 Beef OR 8 Wild game (venison, etc.) Matching Respondents are asked to match responses to a list of items. Example: 1. Match each food to the proper food group by putting the correct lower case letter from the right side in the blank. A. Wheat roll a. Meat and meat products B. Nectarine b. Milk and milk products C. Ham c. Fruits and vegetables D. Yogurt d. Breads and cereals E. Pumpkin e. Sweets F. Oatmeal
12 10 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Close-ended questions with multiple choice answers: Check all that apply This common response format is actually a series of ÒyesÓ or ÒnoÓ items. It is a fast and easy way to obtain such information and also save space. DonÕt make the list too long or the respondents may not consider each item. Examples: 1. What steps have you taken to set up a business? (Check all that apply.) a. Improved product or skills b. Defined product or service c. Identified customers d. Researched market potential e. Filed business name f. Established recordkeeping system g. Applied for resale tax number f. Other 2. What information would you like covered in the next Extension workshop? (Check all choices.) a. Container production b. Landscape design c. Disease control d. Nursery layout e. Weed control practices f. Greenhouse management practices Lists A list provides a series of answers. Respondents may choose one or more depending on the instructions. Examples 1. Listed below are some adjectives which might be used to describe a person. Please indicate for each adjective, whether the adjective does or does not describe you. (Circle one number for each adjective.) Describes Does not DonÕt me describe me know a. Ambitious b. Happy c. Idealistic d. Outgoing From the list provided, select THREE adjectives which best describe you. (Place the letter of the adjective on the lines provided.) 1. a. Ambitious 2. b. Happy 3. c. Idealistic d. Outgoing
13 Q U E S T I O N N A I R E D E S I G N 11 Ranking Rank ordering is a multiple-choice option. Respondents are given various responses and asked to rank them in order of importance or indicate a Òtop three.ó Examples: 1. What would you like to know more about? Select three responses from the list and rank them in order of 1, 2, a. What to eat to look better 2. b. How food affects you 3. c. Weight control d. Health foods e. Physical conditioning through diet f. Vitamins 2. What would you like to know more about? Select three responses from the right hand column and rank them in order of first, second, and third choice. 1. 1st choice a. What to eat to look better 2. 2nd choice b. How food affects you 3. 3rd choice c. Weight control d. Health foods e. Physical conditioning f. Vitamins Formatting the questionnaire After you have selected your questions, youõll need to make a series of decisions about the questionnaire formatñits appearance, length, and the order in which the questions will appear. The questionnaire should be pleasing to the eye and easy to complete. The following guidelines offer some tips to help you put the questionnaire together. Begin with an introduction that includes the questionnaireõs purpose, identifies its source, explains how the information obtained will be used, and assures respondents of confidentiality. In mailed questionnaires, reinforce the points you made in the cover letter. The first questions should be easy, avoiding controversial topics. Write interesting questions that are clearly related to the questionnaireõs purpose. DonÕt use openended or long questions with lengthy answer choices in the beginning of the questionnaire. Address important topics early, rather than late, in the questionnaire. Arrange questions so that they flow naturally. Keep questions on one subject grouped together. Start with general questions and then move to those that are specific. Try to use the same type of question and response throughout a series of questions on a particular topic. For example, donõt needlessly break a respondentõs concentration by using a multiple choice format followed by a yes/no question, followed by an open-ended question. Place demographic questions (age, sex, income level, etc.) at the end of the questionnaire. Print it in an easy-to-read typeface. A numbered response should mean the same thing throughout the questionnaire.
14 12 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Example: If you begin with: 1 No 2 Yes donõt switch to: 1 Yes 2 No Avoid making respondents turn a page in the middle of a question or between a question and answer. Be sure that the question is distinguishable from the instructions and the answers. You could put the instructions in boldface or italics. Dillman (1978) suggests using lower case letters for questions and upper case letters for answers. Questions and answers are easiest to read if they flow vertically. By placing answer choices under questions (rather than side by side), the respondent moves easily down the page. If you feel this format results in too much wasted space, you may wish to reorganize your questions. Example: 1 Excellent 2 Good 3 Fair 4 Poor Rather than: Excellent Good Fair Poor Give directions about how to answer. Include directions in parentheses immediately following questions. It is better to repeat directions too often than not enough. Here are some examples of specific instructions you might use: Circle the number of your choice; circle only one; check all that apply; please fill in the blank; enter whole numbers; please do not use decimals or fractions; etc. Pre-code as many items and response categories as possible to help tabulate and analyze data more quickly. When data is precoded, it can be entered directly from the questionnaire. Try to position the response blanks in the same place on the page to make tabulation easier. Use transitional statements to enhance continuity. Transitional statements serve three functions: 1) to signal that a new topic is about to begin; 2) to start new pages; and 3) to break up the monotony of a long series of questions. Examples: Next, we would like to ask you several questions about the organizations you belong to in your community. Another important purpose of this survey is to learn how you feel about the work of service organizations in your community. Finally, we would like to ask a few questions about you to help us interpret the results. Make sure that the respondent is referring to the same program mentioned in the questionnaire and defining it similarly. A Òvalidation itemó (Bennett, 1982) at the beginning of the questionnaire identifies the program and sets the stage for the questions to follow. It is a brief summary of the programõs activities and the people involved. Example: The Dell County Extension family living program included a variety of activities during 1989 focusing on teaching money management and budgeting skills to help families manage their resources better. These activities included lunch and learn programs, computer budgeting workshops, letter series and short courses. Consumers from across the county attended these activities on Money Management Skills.
15 Filter or screen questions Some questions may not apply to all respondents. For these Òscreen questions,ó make it clear who should answer the question. Also be sure to give directions for those not expected to respond. Dillman (1978) makes three suggestions: 1) use arrows to guide respondents from one question to the next; 2) indent all questions that may be screened; or 3) use boxes to direct respondents past the questions(s) they donõt need to answer. Examples: Q-5 Do you own or rent the home in which you now live? 1 Own home If you own your home, 2 Rent home skip from here to Q-14 on the next page Q U E S T I O N N A I R E D E S I G N 13 (If you rent) Q-6 How much is your monthly rent? 1 Less than $100 2 $100 to $199 3 $200 to $299 4 $300 or more OR Q-5 Do you own or rent the home in which you now live? (Circle the number of your answer.) 1 Own home 2 Rent home (if you own your home) (if you rent your home) Q-6a How much is your monthly house Q-6b How much is your monthly rent? payment (without property taxes)? 1 Less than $100 1 Less than $100 2 $100 to $199 2 $100 to $199 3 $200 to $299 3 $200 to $299 4 $300 or more 4 $300 or more Q-7a How much per month do you Q-7b Which, if any, of these is pay for electricity, garbage included in your monthly rent? collection, heat and water? (Circle all that are included.) 1 Less than $25 1 Electricity 2 $25 to $74 2 Garbage 3 $75 to $124 3 Heat 4 $125 or more 4 None of the above
16 14 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Pretesting the questionnaire Pretesting is an indispensable part of questionnaire design. Many practitioners feel that if the resources to pretest the questionnaire are not available, itõs best to postpone the study until the resources are available. This means you must examine individual questions, as well as the whole questionnaire, very carefully. Allow enough time to incorporate any revisions. Unfortunately, too many people consider pilot testing a perfunctory task if they consider it at all. According to Salant and Dillman (1994), any pretest needs to answer the following questions: Ñ Does each question measure what it is intended to measure? Ñ Do respondents understand all the words? Ñ Are questions interpreted similarly by all respondents? Ñ Does each close-ended question have an answer that applies to each respondent? Ñ Does the questionnaire create a positive impressionñone that motivates people to answer it? Ñ Are the answers respondents can choose from correct? (Are some responses missing? Do some questions elicit uninterpretable answers?) Ñ Does any aspect of the questionnaire suggest bias on the part of the researcher? Ask colleagues to review the questionnaire critically. Let coworkers read the questions to see if the wording and instructions are clear, and if the questionnaire will accomplish the study objectives. Consider reviewersõ comments carefully; then decide if they enhance the questionnaire. Select people as similar to your respondents as possible to pretest the questionnaire. Choose people to represent a cross-section of the population. Simulate the actual data collection procedure as closely as you can,whether it is a mail survey, telephone or direct interview. If youõre using a mail questionnaire, have people answer it without any help and afterwards, ask for their suggestions. In an interview, have the interviewer conduct the pilot test, either by phone or face-to-face, as it will actually be done. Obtain feedback about the form and content of the questionnaire. Were any questions misunderstood? Were the directions clear? Was the questionnaire too long or too difficult? How long did it take to fill out? Was there enough space for responses, etc.? Assess whether the questions produce the information needed to satisfy the studyõs purpose. Try the tabulation and analysis procedures to make sure that the questionnaire yields data that can be analyzed in the way that is needed. Revise. Check the final draft. Go over each question and ask: What will the information obtained from the question mean? How much will it contribute to the study? (Adapted from Rohs, 1985; Sudman and Bradburn, 1986; Salant and Dillman, 1994)
17 Q U E S T I O N N A I R E D E S I G N 15 References Babbie, Earl R. Survey Research Methods. Belmont, CA: Wadsworth Publishing Co., Bennett, Claude F. Reflective Appraisal of Programs (RAP): An Approach to Studying Clientele-Perceived Results of Cooperative Extension Programs. Ithaca, NY: Cornell University Media Services, Dillman, Donald A. Mail and Telephone Surveys: The Total Design Method. New York: John Wiley and Sons, Inc., Fischer, Christy and Jeffrey D. Layman. Edge: Constructing a Questionnaire. Ohio State University: Ohio Cooperative Extension Service, Newsome, Bob W., et al. Northeast Area Evaluation Process. Kansas State University, Kansas State Cooperative Extension Service, n.d. Rohs, F. Richard. Questionnaire Construction. Athens, GA: Cooperative Extension Service, Salant, Priscilla and Don Dillman. How to Conduct Your Own Survey. New York: John Wiley & Sons, Inc Sawer, Barbara J. Evaluating for Accountability. Corvallis: Oregon State University Extension Service, Sudman, Seymour and Norman M. Bradburn. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass Publishers, 1982.
18 Appendix Attribute information When you collect attribute data (personal characteristics such as age, education, income,) it is often a good idea to formulate responses you can compare to existing data; for example, categories from a previous study, or the same categories as reported in U.S. Census data. You can then compare your data to the census to demonstrate how closely your sample reflects the general population. Even if you have no comparisons currently in mind, collecting this type of information about respondents can contribute to a Òdata bank.ó It will allow you to build a Òparticipant profileóthat describes participants in terms of key variables, compares the characteristics of those who participate and those who donõt, assesses the degree to which target audiences are being reached, and plan strategies for promoting future programs effectively. Here are a few tips for collecting such data. (Adapted from Sawer, 1984) Place attribute questions at the end of the questionnaire. Avoid overlapping categories. Use exclusive categories; for example 25Ð34, 35Ð44, 45Ð54, etc., not 25Ð35, 35Ð45, 45Ð55. Ask age questions directly, but be diplomatic. Some respondents may perceive ÒHow old are you?ó as a little blunt. ÒWhat is your age?ósoftens the request. ÒPlease circle the number of the category which includes your ageó is inoffensive. If you need more precisely refined figures, you could ask ÒWhat was your age at your most recent birthday?ó(most people hope they havenõt yet celebrated their ÒlastÓ birthday.) Also avoid asking for a month and/or year of birth unless itõs absolutely necessary, since such information will require many extra calculations. Phrase income questions carefully. Do you want to know the householdõs (or respondentõs) annual income? Monthly income? Net? Gross? Taxable? Take-home pay? Some people are sensitive to income questions. Instead of asking ÒWhat was your total household income in 1995?Ó you could ask, ÒWhich of the following categories best describes your total household income for 1995?Ó Offer a choice of several broad dollar amounts. Use appropriate employment categories. Employment status is one attribute that is hard to describe. The following example is a fairly lengthy one. (The respondent is asked to read the entire list, then select the one response which best describes his/her employment status.) A. Employed full-time in the work force B. Employed part-time in the work force C. Unpaid full-time in farm, family or home business D. Unpaid part-time in farm, family or home business E. Unemployed, seeking work F. Unemployed, not seeking work G. Student, employed part-time H. Student, not seeking work I. Retired, employed part-time J. Retired, not seeking work A simpler breakdown is: A. Employed B. Unemployed, seeking work C. Not in the labor force
19 Q U E S T I O N N A I R E D E S I G N 17 Some examples of attribute questions follow. 1. What is your age? (Circle one number.) 1. Under years years years years years or older 2. Are you married? 1. No 2. Yes 3. How many people live in your household, counting yourself? 4. Are you employed outside the home? 1. No 2. Yes If you work outside the home, do you work: 1 Full-time? 2 Part-time? 5. What is the highest level of education that you have completed? (Circle one number.) 1. Some grade school (1 8) 2. Some high school (grades 9 12) 3. High school graduate 4. Some college or technical school 5. College graduate or beyond 6. Where do you live? (Circle one number.) 1. Rural farm 2. Rural community (less than 2,500 people) 3. Town between ,000 people 4. City between 25,000 50,000 people 5. City over 50,000 people 7. What is your race? (Circle one number.) 1. Anglo or white 2. Black 3. Hispanic 4. Asian 5. Native American 6. Other 8. Is your current agricultural operation a single-family operation, a partnership, a family-held corporation, or a nonfamilyheld corporation? (Circle one number.) 1. Single-family operation 2. Partnership 3. Family-held corporation 4. Non-family-held corporation 5. Other (specify) 9. What was the approximate gross value of farm sales from your operation last year including crops sold, animals sold, and dairy products? (Circle one number.) 1. Less than $2, $2,500 to $4, $5,000 to $9, $10,000 to $19, $20,000 to $39, $40,000 to $99, $100,000 to $199, $200,000 to $499, $500,000 or more 10. In the past year did you work off-the-farm in a part-time or full-time job? (Circle one number.) 1. Did not work off the farm 2. Was employed part-time off the farm 3. Was employed full-time off the farm
20 Author: Ellen Taylor-Powell is a program development and evaluation specialist for Cooperative Extension, University of WisconsinÐExtension. An EEO/Affirmative Action employer, University of WisconsinÐExtension provides equal opportunities in employment and programming, including Title IX and ADA requirements. Requests for reasonable accommodation for disabilities or limitations should be made prior to the date of the program or activity for which they are needed. Publications are available in alternative formats upon request. Please make such requests as early as possible by contacting your county Extension office so proper arrangements can be made. If you need this information in an alternative format, contact the Office of Equal Opportunity and Diversity Programs or call Extension Publications at (608) by the Board of Regents of the University of Wisconsin System doing business as the division of Cooperative Extension of the University of WisconsinÐExtension. Send inquiries about copyright permission to: Director, Cooperative Extension Publications, 201 Hiram Smith Hall, 1545 Observatory Dr., Madison, WI You can obtain copies of this publication from Cooperative Extension Publications, Room 170, 630 W. Mifflin Street, Madison, WI 53703, or by phoning (608) G Program Development and Evaluation Questionnaire Design: Asking Questions with a Purpose RP M-200
D-1369 Questionnaire Design Asking Questions with a Purpose Mary G. Marshall Program Development & Evaluation The Texas A&M University System CONTENTS Constructing a Questionnaire 3 Kinds of Information
G3658-6 Program Development and Evaluation Analyzing Quantitative Data Ellen Taylor-Powell Statistical analysis can be quite involved. However, there are some common mathematical techniques that can make
Questionnaire Design Asking Questions with a Purpose Paul Pope Extension Program Specialist Evaluation Chris Boleman Assistant Professor and Extension Specialist Scott Cummings Assistant Department Head
Capstone Suggestions for Survey Development for Research Purposes 1. Begin by listing the questions you would like to answer with the survey. These questions will be relatively broad and should be based
Designing & Conducting Survey Research Santa Monica College Fall 2011 Presented by: Hannah Alford, Director Office of Institutional Research 1 Workshop Overview Part I: Overview of Survey Method Paper/Pencil
Designing Effective Survey Instruments Thomas M. Archer, Ph.D. 2007 Definitions: Handout Some Terms Data Instrumentation Quantitative and Qualitative Survey and a Questionnaire Reliability and Validity
Innovation Insight Series Number 14 http://www.psu.edu/president/pia/innovation/ Across the University many people use data in assessment and decision-making. Data-based decision-making, an essential element
Survey research Chaiwoo Lee Postdoctoral Associate MIT AgeLab email@example.com agelab.mit.edu Contents Key characteristics of survey research Designing your questionnaire Writing the questions Putting them
Conducting Market Research Using Primary Data Kynda R. Curtis, Ph.D. Assistant Professor and State Extension Specialist Department of Resource Economics, University of Nevada, Reno WEMC FS#7-08 Overview
Custodial Mothers and Fathers and Their Child Support: 2011 Current Population Reports By Timothy Grall Issued October 2013 P60-246 IntroductIon This report focuses on the child support income that custodial
QUESTIONNAIRE DESIGN for Research and Assessment YAO HILL Faculty Specialist The quality of the data will be no better than the most error-prone feature of the survey design. - Floyd J. Fowler, Jr., 2002
PHILANTHROPIC VENTURES FOUNDATION 1222 Preservation Park Way Oakland CA 94612 BILL SOMERVILLE President Proposal Writing Kit Tips & Techniques 1. Writing A Proposal: A Conceptual Framework 2. The Letter
Stafford County Public Schools Apply for Free and Reduced Price Meals OR Prepay for Meals Online! Dear Parent/Guardian, Stafford County Public Schools Is pleased to announce the availability of applying
Using Surveys and Affective Assessments in FCSE Evaluation and Research in FCSE Overview Data important to FCSE rely on more than scores on cognitive (knowledge) based test. FCSE Programs should demonstrate
Identifying Social Attitudes and Barriers to Water Conservation A Community Water Survey H Lane 1, C Krogh 2, L O Farrell 3 1 Eurobodalla Shire Council, firstname.lastname@example.org 2 Eurobodalla Shire
Guide to Effective Staff Performance Evaluations Compiled by Human Resources Siemens Hall, Room 211 The research is clear. The outcome is consistent. We know with certainty that the most powerful leadership
Basic Principles of Survey Design Thomas Lindsay Andrew Sell Introductions Name, Institution, why are you here? Survey Review 5 minutes Political Participation and Attitudes Survey What is the research
S0201 SELECTED POPULATION PROFILE IN THE UNITED STATES 2013 American Community Survey 1-Year Estimates Supporting documentation on code lists, subject definitions, data accuracy, and statistical testing
BUSINESS WRITING: LETTERS, E-MAILS, REPORTS, CASES, AND PRESENTATIONS WHY IS EXCELLENT PROFESSIONAL WRITING SO IMPORTANT TO YOU? Hiring a new employee is expensive, and companies often evaluate their recruiters
HART RESEARCH ASSOCIATES Study #11127--page 1 1724 Connecticut Avenue, NW Interviews: 1,355 adults Washington, DC 20009 Dates: April 8-14, 2014 (202) 234-5570 FINAL Study #11127 48 Male 52 Female 
Project on Improvement of Local Administration in Cambodia What is Training Needs Assessment? Five Steps of Training Needs Assessment Step 1: Identify Problem and Needs Step 2: Determine Design of Needs
Trumbull Career and Technical Center 528 Educational Highway Warren, Ohio 44483 Toll Free 1-866-737-6925 Dear Parent/Guardian: Children need healthy meals to learn. TCTC offers healthy meals every school
Farm Succession Research, Trends and Programs in Wisconsin Joy Kirkpatrick, Outreach Specialist, Center for Dairy Profitability Agenda for the hour Farmer Retirement & Farm Succession Survey Implications
Employment-Based Health Insurance: 2010 Household Economic Studies Hubert Janicki Issued February 2013 P70-134 INTRODUCTION More than half of the U.S. population (55.1 percent) had employment-based health
Wisconsin State FFA Degree Handbook Wisconsin Association of FFA, Inc. Wisconsin FFA Center, Inc. Updated January 2006 1 CONTENTS 1. State FFA Degree Minimum Requirements 2. Applying for State FFA Degree
10 Steps for Planning Educational Programs Guidelines to Help You Complete the Worksheet and Develop Quality Programs 1. Situation Study your county/area to assess the needs for educational programming.
University of Wisconsin-Extension Cooperative Extension Madison, Wisconsin PD &E Program Development & Evaluation Using Excel for Analyzing Survey Questionnaires Jennifer Leahy G3658-14 Introduction You
Unit B: Understanding the Purpose of Record Keeping for Agribusinesses Lesson 2: Student Learning Objectives: Instruction in this lesson should result in students achieving the following objectives: 1.
Appendix 1 Water Quality and Water Usage Surveys This appendix contains copies of the Water Quality Survey and the Lake Usage Survey that we used to complete the watershedbased community assessments. We
Suggestions for Survey Design: Things to consider when conducting a survey Did you know that has conducted over 50 surveys in the past two years? The research office conducts annual surveys of first-time
TAKING NOTES I. There are many reasons for taking lecture notes. A. Making yourself take notes forces you to listen carefully and test your understanding of the material. B. When you are reviewing, notes
Survey Research Claudia J. Stanny PSY 6217 Research Design Writing Questionnaire Items Open-ended items Partially open-ended items Closed (Restricted) items Provide a choice for all potential responses
Facilitating Adult Learning How to Teach so People Learn Prepared by: Dr. Lela Vandenberg Senior Leadership & Professional Development Specialist 11 Agriculture Hall East Lansing, MI 48824-1039 (517) 353-1898
Welcome to today s Coffee Break presented by the Evaluation and Program Effectiveness Team in the Division for Heart Disease and Stroke Prevention at the Centers for Disease Control and Prevention. *Note:
ABC COMPANY EMPLOYEE PERCEPTION SURVEY Prepared By SDS ABC Company,. Page 1 TABLE OF CONTENTS Overview 4 Summary 6 PAGE Conclusions and Recommendations 15 QUESTION BY QUESTION 17 Dept 18 Do you have employees
Managing money online working as well as we think? A behavioural economics study for the Keep Me Posted campaign Prepared by February 2015 About is one of Europe's leading specialist economics and policy
California State University, Sacramento College of Education Multiple Subjects Program Assessment Plan Spring 2001 Contents Multiple Subject Program Group Mission Statement Expected Learning Outcomes (CSTP)
Educational Attainment in the United States: 2003 Population Characteristics Issued June 2004 P20-550 The population in the United States is becoming more educated, but significant differences in educational
G3751 Hiring a Planning Consultant: A Guide to Preparing a Request for Proposals Anna Haines What is a request for proposals? When a community plans to provide a new service or product, it solicits what
RESEARCH REPORT SUBMITTED TO THE OFFICE OF THE PRIVACY COMMISSIONER FOR PERSONAL DATA Survey on Person-to-person Direct Marketing Calls Social Sciences Research Centre The University of Hong Kong August
Performing a Community Assessment 37 STEP 5: DETERMINE HOW TO UNDERSTAND THE INFORMATION (ANALYZE DATA) Now that you have collected data, what does it mean? Making sense of this information is arguably
Fact sheet FS995 For a comprehensive list of our publications visit www.rce.rutgers.edu A Step-by-Step Guide to Developing Effective Questionnaires and Survey Procedures for Program Evaluation & Research
Philosophy Online class observations are meant to facilitate an instructor s professional growth. They will be used to create an opportunity for reflection and stimulate ideas for improvement in the online
PREPARATION GUIDE FOR WRITTEN TESTS Prepared by: The Department of Administrative Services Human Resources Management August 2004 GENERAL INFORMATION ON WRITTEN TESTS Two types of questions are often used
Girl Scouts North Carolina Coastal Pines 6901 Pinecrest Road Raleigh, NC 27613 HOME GROWN IN NORTH CAROLINA PG916W/10-14 North Carolina Department of Agriculture and Consumer Services Patch Program North
Key outcome data analyses Outcome information describes changes and levels of performance for individuals, families, groups, organizations, and communities. These changes may relate to changes in knowledge,
Strategic Communications for Nonprofit Organizations: Seven Steps to Creating a Successful Plan, Second Edition by Sally J. Patterson and Janel M. Radtke Copyright 2009 John Wiley & Sons, Inc. Strategic
4th Annual Celebration of Teaching Excellence at Cornell What Are Students Learning? Tech Tips for Getting and Using Instant Feedback Students arrive in our courses with varying levels of knowledge about
Maximizing Nutrition at Minimal Cost Outline: Get down to the nutrition basics Identify health issues at hand Understand the challenges & barriers Practical application & solutions Rate Your Plate What
Step 1: Establish goals Sept 2: Develop questions and responses Step 3: Pilot test questions Step 4: Re-evaluate each question General Tips A. Identify what you want to learn. Writing Good Survey Questions
GUIDE TO EFFECTIVE STAFF PERFORMANCE EVALUATIONS The research is clear. The outcome is consistent. We know with certainty that the most powerful leadership tool for improving productivity and increasing
HART RESEARCH ASSOCIATES Study #11843--page 1 1724 Connecticut Avenue, NW Interviews: 1200 adults Washington, DC 20009 Dates: April 28-May 10, 2016 (202) 234-5570 FINAL Study #11843 April May 2016 48 Male
HOW TO APPLY FOR FREE AND REDUCED PRICE SCHOOL MEALS Please use these instructions to help you fill out the application for free or reduced price school meals. You only need to submit one application per
Writing Better Survey Questions Mike Pritchard 5 Circles Research 425-968-3883, email@example.com, www.5circles.com, @thatresearchguy Why are we here? Increase competence and consciousness Good questions
Survey of Clinical Trial Awareness and Attitudes Purpose of the Study Over $2 billion per year is spent on recruiting patients for clinical trials, yet slow patient enrollment is the primary cause of delay
Basic Concepts in Research and Data Analysis Introduction: A Common Language for Researchers...2 Steps to Follow When Conducting Research...3 The Research Question... 3 The Hypothesis... 4 Defining the
MILFORD EXEMPTED VILLAGE SCHOOL DISTRICT Nutrition Services 777 Garfield Avenue Milford, OH 45150 (513) 831-5030 **NOW AVAILABLE** ONLINE FREE AND REDUCED APPLICATIONS FOR MILFORD EXEMPTED VILLAGE SCHOOLS
TRAINING NEEDS ANALYSIS WHAT IS A NEEDS ANALYSIS? It is a systematic means of determining what training programs are needed. Specifically, when you conduct a needs analysis, you Gather facts about training
HART RESEARCH ASSOCIATES Study #11843--page 1 1724 Connecticut Avenue, NW Interviews: 1200 adults nationally, and 303 adults in the City of Washington, DC 20009 and 300 adults in the suburbs (202) 234-5570
Appendix III: Complete Set of Frequencies The following pages contain a complete set of survey frequencies. The number of respondents for each question is 1,172 unless noted otherwise. Question B How long
This document presents an overview of some common issues that nonprofits face when constructing a survey. While there are certainly different ways to structure and administer a survey, these issues apply
Fit With Fiber Graphing Cereal Skills: Science and Math Objective: Students gather and graph information about favorite breakfast cereals. Background Fiber is present in all plants that are eaten for food,
These exercises look at the topics in the context of a communications mix. We start with an examination of what advertising objectives are (Exercise 7.1). We then look at how to set advertising objectives
JOB ANALYSIS AND DESCRIPTION 1. Importance of Job Analysis and Description Job analysis and job description are closely interrelated. The information gathered on each job through job analysis is subsequently
HOW TO LEVERAGE MAXIMUM INFORMATION FROM YOUR PATIENT SURVEY By Keith Meadows Understand the BIG picture Craft a good question GET THE LAYOUT & SEQUENCING RIGHT START AT THE END CHOOSE THE QUESTION TYPE
ALE Paper Title: Terra Incognita: An Exploration of Leadership Skills, Public Issues, and Program Evaluation This paper explores the similarities and differences in evaluation of leadership/public issues
PB 1540 Establishing and Using a Farm Financial Record-Keeping System Delton C. Gerloff, Associate Professor, Agricultural Economics Robert W. Holland Jr., Assistant Area Specialist - Farm Management For
Evaluating the effectiveness of Reconciliation Action Plans Report prepared by Auspoll REPORT PREPARED BY: David Stolper Senior Research Partner firstname.lastname@example.org T/ 02 9258 4462 Nick Wyatt Consultant
L-2380 12-99 Career Planning for Teens Teens today have a world of opportunities to consider for their careers. According to the Texas Workforce Commission, Texas has more than 29,561 separate occupations.
You re One in Seven Billion! We ve all heard the expression, You re one in a million!. With the ever-growing number of people on the planet, it might be more accurate to say, You re one in seven billion!
Using the American Community Survey Data The ACS website is accessible from www.census.gov/acs. In the middle of the screen choose In-depth Data Go directly to the ACS Datasets Tab The ACS Datasets Tab
Primary Data Collection Methods: Survey Design Prepared by Yann Duval, UNESCAP ARTNeT Capacity Building Workshop on Trade Research Bangkok, Thailand, 22-25 March 2005 Primary Data Collection Primary data
Overview Summary Background Information Communities Across America Today By Sarah Glasscock Paz Gonzalez, the matriarch of the fictional Gonzalez clan, lives in San Antonio, Texas. As she strolls through
A BUSINESS PLAN IS IMPORTANT WHEN WORKING WITH A LENDER You don t have to borrow money to expand a business and make your dreams come true, but borrowing money can make those dreams come true sooner than
PHASE II Choosing The Right Evaluation Questions Writing well thought-out, relevant evaluation questions is the secret to a meaningful evaluation. Keep in mind that the more specific the focus of the evaluation,
Toplines The Kaiser Family Foundation Health Insurance Survey October 2004 Methodology The Kaiser Family Foundation Health Insurance Survey was designed and analyzed by researchers at the Kaiser Family
Appendix B. National Survey of Public School Teachers This survey is based on a national random sample of 1,010 K 12 public school teachers. It was conducted by mail and online in fall 2007. The margin
Guide to Effective Staff Performance Evaluations HRS Human Resource Services-Operations The research is clear. The outcome is consistent. We know with certainty that the most powerful leadership tool for
Introduction to Social Marketing for Water Outreach NH Water and Watersheds Conference 2011 March 25 26, 2011 Plymouth State University, NH Julia Peterson NH Sea Grant and UNH Cooperative Extension Julia.email@example.com
Agriculture, Food & Natural Resources Career Cluster Agricultural Animal Production and Management Course Number 01.43200 Course Description: The goal of this course is to provide all students instruction
C A L H O U N COUNTY SCHOO LS Dear Parent/Guardian: Children need healthy meals to learn. Calhoun County Schools offers healthy meals every school day. Breakfast costs $1.50; lunch costs $1.75. Your children
COMMUNITY COLLEGE COMPRESSED CALENDARS: RESULTS OF A STUDENT SURVEY AND A FACULTY SURVEY 1 Michael Carley 2 Porterville College Abstract Many community colleges are considering changes in their traditional
401(k) Participant Study: 2015 August 2015 Health insurance and 401(k) plans are the two must have benefits for people looking for a job Benefit/work option must-have if looking for a new job Would think