Technical Report. For this research Ipsos MORI interviewed a sample of 1,005 adults Wales between the 5 th and 9th March 2011.

Size: px

Start display at page:

Download "Technical Report. For this research Ipsos MORI interviewed a sample of 1,005 adults Wales between the 5 th and 9th March 2011."

Edgar Stanley
7 years ago
Views:

1 Technical Report This technical report, which can be appended to the Electoral Commission s Post NAW Referendum 2011 topline, contains a summary of the technical details and statistical reliability of this quantitative research project. 1. Methodology For this research Ipsos MORI interviewed a sample of 1,005 adults Wales between the 5 th and 9th March Interviews were conducted by telephone using Random Digit Dialling. We interviewed a representative quota sample. Quotas were set by age, gender, working status and region. The data were then weighted to the known national population profile in Wales. 2. Statistical Reliability The nature of research methodology dictates that respondents to the questionnaire are samples of the total population of Wales so we cannot be certain that the figures obtained are exactly those we would have if everybody in Wales had been interviewed (the true values). However, the variation between the sample results and the true values can be predicted from the knowledge of the size of the samples on which the results are based, the representativeness of the sample itself and the number of times that a particular answer is given. The confidence with which this prediction can be made is usually chosen to be 95% - that is, the chances are 95 in 100 that the true value will fall within a specified range. Given that we have both set quotas during the fieldwork phase as well as weighted our data to be representative of the profile of Wales, this reduces the effective base size from 1,005 to All statistical reliability has been calculated using this effective base size. Effective base:when data are weighted, it creates this 'design effect'which can negatively impact on the reliability of the data. Therefore the effective base size (the base size left when taking this into account) is the base size that can be used for significance testing. This is also known as the design effect, wherein some factors of the research methodology can negatively impact on the reliability of the data. In this case, this is due to the weighting used. 1

Interviews were conducted by telephone using Random Digit Dialling. We interviewed a representative quota sample. Quotas were set by age, gender, working status and region.

2 The table below illustrates the predicted ranges for different sample sizes and percentage results at the 95% confidence interval. Size of sample on which survey result is based Approximate sampling tolerances applicable to percentages at or near these levels 10% or 90% 30% or 70% 50% interviews interviews interviews interviews interviews interviews interviews interviews ,000 interviews For example, with an effective base size of 600 where 50% give a particular answer, the chances are 19 in 20 that the true value (which would have been obtained if the whole population had been interviewed) will fall within the range of +4 percentage points from the sample result (i.e. between 46% and 54% inclusive). When results are compared between separate groups within a sample, different results may be obtained. The difference may be real, or it may occur by chance (because not everyone in the population has been interviewed). To test if the difference is a real one i.e. if it is statistically significant we again have to know the size of the samples, the percentage giving a certain answer and the degree of confidence chosen. If we assume 95% confidence interval, the differences between the results of two separate groups must be greater than the values given in the table below. 2

8 8 200 interviews 5 7 7 300 interviews 4 6 6 400 interviews 3 5 5 500 interviews 3 4 5 600 interviews 3 4 4 700 interviews 2 3 4 1,000 interviews 2 3 4 For example, with an effective base size of

3 Size of samples compared Differences required for significance at or near these percentage levels 10% or 90% 30% or 70% 50% and and and and and and and and and and and and and 1, and 1, So, for example, if we were to compare voters and non-voters views in the Electoral Commission s Post NAW Referendum Research in Wales, we would be comparing sub group samples of approximately 624 (voters) and 381 (non-voters). With these base sizes, where (e.g.) 30% of voters give a particular answer and (e.g.) 37% of non-voters give the same answer, the chances are 19 in 20 that the difference in responses between voters and non-voters is a true (statistically significant) difference (because the difference is seven or more). 3. Interpreting data tables When looking at the data tables for this research, you will notice that in addition to the percentage figures, there are also some small letters and other indicators within the text on the tables. We have explained these in full below. Letters assigned to percentage figures: if you look at the cross breaks (the columns running across the top of each table, e.g. age, gender etc), each cross break is assigned a letter (written in lower case directly underneath the column heading) that it has throughout the tables. For example, total is x', voter is 'a', non-voter is 'b'and male is 'c'. If a letter is written alongside a percentage figure within the body of the table, it is highlighting the fact that the percentage is statistically significantly higher than the corresponding figure in column 'a'or 'b'or 'c'etc. For example, if the percentage in the total column (x) has 'b'and 'c'written alongside it, then the total (x) percentage is significantly higher than that of non-voters (b) and males (c). 3

for example, if we were to compare voters and non-voters views in the Electoral Commission s Post NAW Referendum Research in Wales, we would be comparing sub group samples of approximately 624

4 A practical example of this from the tables can be found in question 4. In the total column (column x) it is shown that 59% of respondents always vote at elections for the Welsh Assembly. As the table shows, this figure is significantly higher than columns b (non-voters 25%), e (18-24 year olds 25%) and g (35-44 year olds 47%). All other columns either have no significant difference with the total, or are significantly higher than the total column themselves (which would be shown by a 'x'next to the percentage in said column). Small base size: This is a base that is too small to statistically analyse through significance testing. The small number of respondents means that we cannot guarantee representativeness and that the findings are therefore indicative. We would thus advise that you not use these percentages for statistical analysis but that they are used indicatively with a note to the reader outlining the small base size and its limitations. 4. Social Grade Definitions A Professionals such as doctors, surgeons or dentists; chartered people like architects; fully qualified people with a large degree of responsibility such as senior editors, senior civil servants, town clerks, senior business executives and managers, and high ranking grades of the Services. B C1 C2 D E People with very responsible jobs such as university lecturers, matrons of hospitals, heads of local government departments; middle management in business; qualified scientists, bank manages, and upper grades of the Services, police inspectors. All others doing non-manual jobs: nurses, technicians, pharmacists, salesmen, publicans, people in clerical positions and middle ranks of the Services, police sergeants. Skilled manual workers/craftsmen who have served apprenticeships; foremen, manual workers with special qualifications such as long distance lorry drivers, security officers and lower grades of Services, police constables. Semi-skilled and unskilled manual workers, including labourers and mates of occupations in the C2 grade and serving apprenticeships; machine minders, farm labourers, bus and railway conductors, laboratory assistants, postmen, waiter/waitress, door-to-door and van salesmen. Those of lowest levels of subsistence including pensioners, casual workers, the unemployed and other with minimum levels of income. 4

All other columns either have no significant difference with the total, or are significantly higher than the total column themselves (which would be shown by a 'x'next to the percentage in said

5 5. Welsh Assembly region definition 5

6 6. Claimed level of turnout As part of this research respondents were asked whether or not they had voted in the referendum. The chart below shows that 60% of respondents said they did vote, compared to the official turnout of 35%. This finding of overclaimed turnout is common in this type of research and in 2010 we found that 82% said they had voted in the General Election. Q As you may know, there was a referendum in Wales on Thursday 3rd March about the powers of the National Assembly for Wales. Many people have told us they didn t manage to vote in the referendum on 3rd March. How about you did you manage to vote in the referendum? 2011 Referendum Yes No 2010 General Election 49 18% 40% 60% 82% Base: 1,005 adults, fieldwork dates: 5 th March 9 th March 2011 Source: Ipsos MORI 6

This finding of overclaimed turnout is common in this type of research and in 2010 we found that 82% said they had voted in the General Election.

The Office of Public Services Reform The Drivers of Satisfaction with Public Services

The Office of Public Services Reform The Drivers of Satisfaction with Public Services Research Study Conducted for the Office of Public Services Reform April - May 2004 Contents Introduction 1 Executive