Same Question but Different Answer Experimental Evidence on Questionnaire Design s Impact on Poverty Measured by Proxies

Transcription

1 Same Question but Different Answer Experimental Evidence on Questionnaire Design s Impact on Poverty Measured by Proxies Joint work with Talip Kilic Thomas Pave Sohnesen Poverty & Inequality Group, Development Research Group, World Bank Visiting Professor Copenhagen University Bonn April, 2015

2 Motivation and research questions Motivation We would like to track poverty more often, but data collection is costly. Predicting poverty using welfare monitoring surveys or other already existing surveys is an approach already in use, that can reduced data collection cost. Malawi s (bad) experience using welfare monitoring surveys to track poverty. Research questions Do people answer identical questions asked in two different questionnaires the same way? Spoiler alert they don t! Does any such variation significantly impact poverty estimated through proxies?

3 Data: Experiment design and sample Experiment is an add on to large living standard survey in Malawi. All households are interviewed twice; with a long and a short questionnaire. Half of sample received short first, the other half received long first. Around 3 months between the two visits Experiment is two extra pages with questions normally only found in long version added to short questionnaire just after household roster for a random set of households. Randomization was successful on observable characteristics. A sample of 2822 households, of which 765 received the experiment.

4 Do answers differ between survey instruments? Number of questions in experiment Number of questions with significant different reporting in long and short questionnaires Binary Food 22 7 Non-Food Shocks 23 6 Ordered Categorical Housing 4 1 Subjective Welfare 4 2 Durable Assets 4 2 Continuous Cell Phone Expenditures 1 1 TOTAL 83 33

5 Do answers differ systematically between survey instruments? Pooled probit regressions of all dummy questions Variable Yes/no dummy (1) (2) (3) Model Specification Household controls NO YES YES Interviewer Fixed Effects NO NO YES Dummy short questionnaire 0.027*** 0.023*** 0.022*** (0.005) (0.005) (0.005) Observations 197, , , percentage point is equivalent to a marginal effect of 9 percent at the mean. The impact is somewhat heterogeneous across groups of questions (food, non-food, shocks), but significant for all types. The impact is heterogeneous across types of households, hence an impact on both level and distribution of predicted consumption can be expected.

6 Do answers differ systematically between survey instruments? Impact for ordered categorical questions more mixed. No significant reporting differences for two housing questions filled in by enumerators. In 5 of 6 questions with a significant different reporting, the population in the short questionnaire report lower numerical values (usually category 1 or 2) In 5 of 6 questions with a significant different reporting, the population in short questionnaire report worse outcomes.

7 Predicting poverty We predict consumption and poverty for all households using Elbers et al. (2003). We establish prediction model using previous round of household consumption survey. We predict 4 different models: 1. Using experiment variables only (variables in two additional pages) 2. Using experiment and non experiment variables 3. Using government preexisting prediction model 4. Non-experiment variables only

8 Predicting poverty Differences in Headcount Poverty Rate Predictions between Long and Short Questionnaires Prediction Model Same population Same households 1. Experiment Only Experiment & Non-Experiment WMS Model Non-Experiment Only

9 Takeaways The different survey instruments do impact what we measure even for identical questions. This experiment cannot : Identify if one questionnaire capture more accurate data than the other. Identify the cause of the observed differences, hence we don t know how to remedy the problem. We do not observe larger reporting differences in later modules, so time in-it-self does not seem to drive results. Similar effects, could also be imagined for impact studies or other survey based research that rely on different questionnaires. Until we understand the challenge better, we suggest piloting both base and proxy questionnaires simultaneously, with enough power to test for survey design impact.

10 Thank you