SUTROFOR Erasmus Mundus Faculty of Agriculture University of Padova - Italy Questionnaire design: an introduction

Transcription

1 SUTROFOR Erasmus Mundus Faculty of Agriculture University of Padova - Italy Questionnaire design: an introduction Mattia Cai

2 Outline Introduction Two examples: Study of mushroom collection in Finland (Cai, Vidale and Pettenella 2009) The World Bank s Living Standards Measurement Study (LSMS) Some issues with designing questionnaires

3 Introduction Survey A means of collecting quantitative information from a particular population about a particular topic Questionnaire A tool frequently used in surveys to collect quantitative information

4 The mushroom study 1/3

5 The mushroom study 2/3 Dalla Valle OY s mushroom business: People pick mushrooms in Finnish forest and sell them to Dalla Valle Oy at collection points Processing and packaging Sell to Italian food importers

6 The mushroom study 3/3 Aims: What are the socio economic characteristics of mushroom pickers? How important is mushroom collection as a source of income for pickers? Data collection: questionnaire administered to pickers at collection points

7 LSMS 1/4 The Living Standards Measurement Study (LSMS) was established in 1980 to improve the type and quality of data collected by statistical offices in developing countries. Objective: collect household data that can be used to ( ) evaluate the effects of various government policies on the living conditions of the population (Grosh and Glewwe 1995). Most data freely available online:

8 LSMS 2/4 Multi-topic survey: consumption, income, savings, employment, health, education, fertility, nutrition, housing, migration, Typically, LSMS surveys consist of three questionnaires: Household questionnaire information on household members Community questionnaire community leaders area asked about community infrastructure Price questionnaire market vendors asked about prices

9 LSMS 3/4 Household questionnaire - structure

10 LSMS 4/4 Household roster

11 Some issues with designing questionnaires

12 What is the objective of your study? What is the group of individuals (population) that you want information about? What kind of information do you want to collect?

13 Types of survey Mail cheap, low response rate, standard cheap, low response rate, standard Telephone: medium cost, medium response rate, standardization depends on interviewer Face to face: Expensive, high response rate

14 Some strategies for higher response Incentive vs. no incentive Short vs. long Colored vs. white Follow-up vs. no follow-up University vs. other Logical flow vs. no logical flow More interesting vs. less interesting

15 Getting the questions right Question wording Questions need to be short, clear, simple and precise Use everyday words Poorly written questions lead to ambiguity and misunderstandings and can be wasteful Responses open, closed, what type of response set will you use?

16 Vague questions How much time do you spend on average for collecting mushrooms in the forest? Less than 30 minutes Between 30 minutes and 1 hour Between 1 and 2 hours More than 8 hours

17 Recall bias Answers to questions that ask respondents to recall past events are often inaccurate because of faulty memory. How many days do you spend on mushroom collection? (average over the past 3 yrs) Weekly ( ) Monthly ( ) Annually ( )

18 Framing effects Even minor changes in wording can change a survey s outcome. Be careful how you frame your questions; aim for neutral terms. Example: Do you think the government is spending too much on assistance to the poor? 13% Yes Do you think the government is spending too much on welfare? 44% Yes

19 Filter questions Filter questions are useful to ensure respondents only answer relevant parts of questions.

20 Ordering the questions Sequence should be logical to the respondents and flow smoothly from one question to the next Questions tend to flow from: general to specific impersonal to personal easy to difficult

21 Problems with the answers Individuals vary in their interpretations of categories. For example Highly probable ~ 60% to 99% Usually ~15% to 99% Rather likely ~1% to 75% Cannot be excluded ~7% to 98%

22 Answers: Likert scale Rate agreement with a series of statements. To what extent do you agree with each of the following statements Strongly agree Agree Neither Disagree Strongly disagree

23 Answers: Likert scale How many steps/boxes should there be? Five to seven response categories is ideal People averse to extreme ends of a scale Avoid absolutes (e.g. almost always vs always; almost never vs never) Add throw away categories at either end.

24 Order effects People may be more likely to endorse first or last response Preceding questions may influence responses to questions that follow

25 Pilot When you have a final draft of your questionnaire: Discuss it with your adviser/colleagues Try to see how it works with a pilot group. Evaluate and modify based on pilot