Labour Force Survey s Electronic Collection Collection strategy, test results and transition Anne-Marie Côté and Guy Laflamme, Statistics Canada September 4 th, 2014
Outline A. Canadian LFS particularities B. Electronic Questionnaire (EQ) collection strategy C. The pilot test: 1. Objectives 2. Design 3. Collection results 4. Data quality 5. Feedback D. Transition and things to come 2 Statistics Canada Statistique Canada
A. Canadian LFS particularities Mandatory Monthly cycle Short collection period (7-10 days) Data released 3 weeks after reference week Sampling unit is the address and all people living in the household are surveyed Once a household is sampled, it has to complete LFS for 6 months (rotate 1/6 of dwellings each month) 3 Statistics Canada Statistique Canada
A. Canadian LFS particularities Sample size: 56 000 households Currently only CATI or CAPI 1st month interview is longer to establish respondents profile Average length of interview 1st month: 13 minutes Following months: 9 minutes Response rates: 90% 1 st month: CATI : 84%, CAPI: 91%, total: 89% Subsequent: 90% 4 Statistics Canada Statistique Canada
B. EQ collection strategy Design Use in-house tool to generate electronic questionnaires (EQGS) Same wording of questions as CATI/CAPI Mostly a paging design with basic edits Collection 1st month will still be CATI or CAPI At the end of each CATI/CAPI interview Verify eligibility If eligible, EQ is offered If respondent agrees, email address is collected 5 Statistics Canada Statistique Canada
B. EQ response process (invitation email) 6 Statistics Canada Statistique Canada
B. EQ response process (login) 7 Statistics Canada Statistique Canada
B. EQ response process (questionnaire) 8 Statistics Canada Statistique Canada
B. EQ response process (questionnaire) 9 Statistics Canada Statistique Canada
B. EQ response process (questionnaire) 10 Statistics Canada Statistique Canada
B. EQ response process (questionnaire) 11 Statistics Canada Statistique Canada
C. Test objectives Primary: Confirm we can do EQ with no changes to release schedule (3 weeks after reference week) Impact on collection operations and systems Impact on processing system Impact on data quality Secondary: Take-up rate, attrition Respondent reaction Mode effect 12 Statistics Canada Statistique Canada
C. Test design Collection: March-June 2013 Parallel test (data not used in production) 6,400 households 3,200 births in March + 3,200 in April Sample not representative of entire country, mainly urban areas Guiding principal: simulates real LFS Mandatory During LFS week to avoid/control for week effect Interviewer cannot identify test cases Selected from the LFS frame Same clusters as production increase statistical power 13 Statistics Canada Statistique Canada
C. Test results: summary Systems can support EQ No measurable impact on data quality No impact on response rate able to convert EQ non-response to CATI response 20% (+) of respondents used EQ Email reminders are effective No statistical evidence of mode effect 14 Statistics Canada Statistique Canada
C. Eligibility, acceptance, and completion Month Sample size Resp. Eligible (eligible/resp.) Accepted Offer (among eligible) EQ Resp. (among accept.) EQ resp. (among all resp.) March 1/2 2,803 73% 58% n/a n/a April full 5,651 72% 60% 44% 9% May full 5,729 73% 60% 50% 21% June full n/a n/a n/a 59%** n/a 15 Statistics Canada Statistique Canada
C. Accepting and responding Accepters Higher education, larger households, younger respondents, employed, geography Online respondents Higher education, larger households, employed, geography Most EQ respondents complete the questionnaire in one session 16 Statistics Canada Statistique Canada
17 Statistics Canada Statistique Canada
C. Retain early adopters and convert Retain early adopter 75-80% of respondents who accepted offer in previous month will accept it for current month 75% of EQ respondent for previous month will complete the questionnaire online in current month Convert When a non respondent becomes eligible, 57% will accept offer When not eligible becomes eligible, 48% will accept offer When refused offer, only 14% will accept it later 18 Statistics Canada Statistique Canada
C. Data quality Estimates (employment, unemployment, distribution by industry and occupation) Edit and imputations Changes in labour force status (gross flows) Question distributions Final variable distributions Respondents behavior Do EQ respondents cheat? Changes in household size Short path More please specify : wording problem? More mark all that apply by design Not more rounding 19 Statistics Canada Statistique Canada
C. Mode effect What is it? Impact of mode on answer Variable specific Focussed on those who accepted the offer Compared those who responded by EQ to CATI 3 perspectives: Compared estimates Propensity score adjustment to address selection bias Regression analysis No measurable mode effect 20 Statistics Canada Statistique Canada
C. Qualitative feedback From technical support, respondent comments, and interviewer debriefings The comments were related to: Offer/invitation wording (too long, online??) EQ and confidentiality Login process & passwords Short collection period Survey-specific details (misspelled names, question concept ) Interviewer training 21 Statistics Canada Statistique Canada
D. Transition to production No embedded experiments Transition Starts in November 2014 For new households (births), verify eligibility, offer EQ not offered to new households in December Ensure that there are no problems One rotation per month from January 2015 on Protocol 4 EQ days: Sunday 2:00 Wednesday 24:00 Email reminders: Monday 14:00 + Wednesday 9:00 22 Statistics Canada Statistique Canada
Future Improve/simplify access (offer, invitation email, access key, password ) Interviewer training Monitor transition and on-going Other models for mode effect? 23 Statistics Canada Statistique Canada
For more information, Pour plus d information, please contact: veuillez contacter : Guy Laflamme and Anne-Marie Côté guy.laflamme@statcan.gc.ca anne-marie.cote@statcan.gc.ca 24 Statistics Canada Statistique Canada