How reliable are household surveys?

Transcription

1 How reliable are household surveys? Phil Bunn, Gareth Anderson, Alice Pugh, Arzu Uluc Bank of England Household Wealth Data and Public Policy Conference, 9-10 March

2 Motivation Need to understand how reliable surveys are if using them to inform policy decisions Evaluate the Bank s own household survey, run by NMG Consulting 2

3 Outline Comparing household surveys to each other and aggregate data: income and mortgage debt What have we learnt about methodology from running the Bank of England/NMG survey: moving online 3

4 Previous literature IFS work shows consumption has been increasingly under-recorded in the Living Costs and Food Survey, (Banks and Johnson (1998) and Barrett et al (2013)) Previous Bank work looked at reliability of balance sheet data in the British Household Panel Survey, (Redwood and Tudela (2004)) Our focus is on balance sheet data and financial distress 4

5 Surveys considered Survey Annual sample Main focus Family Resources Survey 25,000 Income Living Costs and Food Survey 5,000 Consumption Understanding Society/ BHPS 25-30,000/ 5000 Wealth and Assets Survey 20-30,000 per 2 year wave Social & economic change Wealth Bank of England/NMG Survey 2-6,000 Balance sheets/ Attitudes to spending 5

6 Bank of England/NMG Survey Annual survey since 2004, run during September Data on Bank website and annual Quarterly Bulletin article Run online since 2012 Now covers 6,000 households a year, previously 2,000 Focus on balance sheets and attitudes to spending Main benefits for us are timeliness and ability to add bespoke questions 6

7 A measure of gross cash income Employee wages and salaries + Self-employment income + Government benefits + Private pension payments + Investment income = Total gross cash income 7

8 Average household income Annual income per household, ONS aggregate FRS LCFS USoc/BHPS NMG WAS ( )

9 Distribution of individual incomes (2011) Percentage of income 30% HMRC USOC 2011 WAS % 20% 15% 10% 5% 0% Lower value of income band 9

10 Income coverage by component (2011) FRS LCFS Usoc WAS Percentage coverage relative to aggregate data Wages and salaries Self employment Benefits Investment income 0 10

11 Reliability of income data: summary Average income in surveys about 20% lower than implied by aggregate data, except WAS, which is in line Degree of under-recording fairly constant over time WAS better captures high debt to income households, although also has higher investment income NMG survey 5-10% below other household surveys 11

12 Average mortgage debt ONS aggregate FRS LCFS USoc/BHPS NMG WAS Outstanding mortgage debt per household,

13 ONS aggregate FRS LCFS USoc/BHPS NMG WAS ONS less BTL estimate Average mortgage debt Outstanding mortgage debt per household,

14 Mortgage debt to income distribution (2012) Percentages of mortgagors 35 FRS LCFS NMG WAS ( ) < >5 Mortgage debt to income ratio 0 14

15 Reliability of mortgage data: summary All surveys under-record growth of mortgage debt since early 2000s Some, but not all, of that reflects buy-to-let lending Differences in share of high debt households quite small between surveys, although proportion is a bit higher in NMG survey 15

16 What have we learnt moving the NMG survey online 16

17 Moving the NMG survey online What did we do? Measurement of financial distress Measurement of other financial variables Response rates Who has internet access? Bespoke questions 17

18 Moving the NMG survey online 2 online pilots in 2010 (500 households) and 2011 (1000 households) alongside full face to face surveys (2000 households) Full survey online from 2012 (4000 households) and face-to-face parallel run of small number of questions (2000 households) Increased sample size from 2,000 to 6,000 by

19 Mortgage payment problems Percentages of mortgagors having problems paying for accommodation 25 BHPS/USOC NMG face to face 20 NMG online

20 Mortgage arrears Percentage of mortgages in arrears for at least 3 months 6 CML aggregate data BHPS NMG WAS

21 Average income and mortgage debt Online 2012 Face-to-face Mean gross annual income 31,524 33,228 Mean mortgage debt (for mortgagors) 85,189 84,004 21

22 Income distribution Mortgage debt distribution Percentages of households 20 Percentages of mortgagors 12 NMG online NMG face-to-face FRS NMG online NMG face-to-face FRS Annual income (, thousands) < Mortgage debt (, thousands)

23 Response rates Proportion of refusals/don t knows 2012 Online Face-to-face Income question 9% 35% Mortgage debt question 5% 29% Mortgage payment problems question 3% 7% 23

24 Household with internet access By age Percentages of households 100 By income Percentages of households Annual income (, thousands) 24

25 Impact of a 1pp rise in interest rates Borrowers vs savers By age Impact on post-tax income Impact on consumption via cashflow effect Per cent 4 Impact on post-tax income Impact on consumption via cashflow effect Per cent Mortgagors Unsecured borrowers Savers All All -2 25

26 Benefits of moving the NMG survey online Cheaper/easier to administer More accurate reporting of financial distress, not much effect on other variables Higher response rates Easier to include more innovative questions Downside is that not all households have internet access and no opportunity to clarify questions 26

27 Conclusion Household surveys appear reasonably, but not perfectly, reliable. Surveys under-represent high income households, except WAS All surveys under-record the growth of mortgage debt NMG survey broadly follows other surveys. Does less well at margin but is better at measuring financial distress There are advantages to conducting household surveys online experience with the NMG survey has been positive 27