EAA492/6: FINAL YEAR PROJECT DEVELOPING QUESTIONNAIRES PART 2 A H M A D S H U K R I Y A H A Y A E N G I N E E R I N G C A M P U S U S M

Transcription

1 EAA492/6: FINAL YEAR PROJECT DEVELOPING QUESTIONNAIRES PART 2 1 A H M A D S H U K R I Y A H A Y A E N G I N E E R I N G C A M P U S U S M

2 CONTENTS Reliability And Validity Sample Size Determination Sampling Designs Descriptive And Inferential Statistics Pilot Study 2

3 CHOICE OF QUESTIONNAIRES Adopt the questionnaire based on previous studies 3 Adapt the questionnaire based on previous studies Create a NEW questionnaire

4 RELIABILITY AND VALIDITY 4 Validity can be defined as the extent to which any measuring instrument measures what it is intended to measure. Reliability concerns the extent to which an experiment, test or any measuring procedure yields the same results on repeated trials.

5 VALIDITY 5 Validity is the extent to which any measuring instrument measures what it is intended to measure. Validity is about interpretation of data arising from a specified procedure. It is not a test! Thus, validity is not about the measuring instrument itself but the measuring instrument in relation to the purpose for which it is being used.

6 VALIDITY Three types of validity: (1) Content Validity (2) Criterion-Related Validity (3) Construct Validity 6

7 Content Validity 7 Content validity depends on the extent to which an empirical measurement reflects a specific domain of content. Example: A test in arithmetical operations would not be valid if the test problems focused only on addition and neglecting subtraction, multiplication and division. Thus a researcher must be able to specify the full domain of content that is relevant to the particular measurement situation. Example: We must specify all the words that a standard four student should know how to spell. Choose at random the number of words that should be sampled. Then, they must be put in a form that is testable.

8 Content Validity Some limitations 8 The process of determining the domain of the content is more difficult and complex when dealing with the abstract concepts typically found in the social sciences. There is no agreed upon criterion for determining the extent to which a measure has attained content validity. Thus, a measure can only be considered as strongly or weakly valid (i.e., the alternative is not between fully valid or fully invalid measures).

9 Criterion-Related Validity Also known as Predictive Validity 9 Has the closest relationship to what is meant by the term validity Definition: Is an issue when the purpose is to use an instrument to estimate some important form of behaviour that is external to the measuring instrument itself, the latter being referred to as the criterion Example: We assess the validity of college board examination by showing they accurately predict how well high school college seniors will do in college instruction

10 Criterion-Related Validity 10 Example: We validate a written driver s test by showing that it accurately predicts how well some group of persons can operate an automobile. The indicator between the test and the criterion is usually estimated by the size of the correlation.

11 Criterion-Related Validity 11 Have been used mainly in psychology and education. Should be used in any situation or area of scientific enquiry in which it makes sense to correlate scores obtained on a given test with performance on a particular criterion or set of relevant criteria. For some cases, criterion-related validity cannot be used because we cannot determine the relevant criterion variables.

12 Construct validity 12 It is concerned with the extent to which a particular measure relates to other measures consistent with theoretically derived hypotheses concerning the concepts (or constructs) that are being measured. Example: Suppose a researcher wanted to evaluate the construct validity of a particular measure of self-esteem say Rosenberg s self-esteem scale. Theoretically, Rosenberg has argued that a student s level of self-esteem is positively related to participation in school activities. Determine correlation between Rosenberg s self-esteem scale to a group of students and their extent of involvement in school activities. If correlation is positive and substantial, then it supports one piece of evidence on the validity of Rosenberg s self-esteem scale.

13 Involves three steps: Construct validity 13 (1) Theoretical relationships between the concepts must be specified. (2) Empirical relationships between the measures of the concepts must be examined. (3) Empirical evidence must be interpreted in terms of how it clarifies the construct validity of a particular measure.

14 RELIABILITY 14 Reliability concerns the degree to which results are consistent across repeated measures Basic formulation of measurements where X is the observed score, t is the true score and e is the random error

15 RELIABILITY Assumptions: Note : For (3), it is assumed that two sets of measurements are observed for a single person for a single variable.

16 RELIABILITY Therefore, 16 From Assumption (1), This result is true for repeated measurements of a single variable for a single person.

17 RELIABILITY 17 Reliability refers to the consistency of repeated measurements across persons rather than within a single person. Thus, look at the variance of the measurement. Hence

18 RELIABILITY Thus the ratio of true to observed variance is called the reliability of X as a measure of T. 18 Reliability can also be expressed as

19 RELIABILITY 19 The estimate of a measure s reliability can be obtained by correlating parallel measurements. Two measurements (X and X ) are defined as parallel if they have identical true scores and equal variances as shown below:

20 RELIABILITY 20 Thus, Thus it follows that the estimate of reliability is simply the correlation between parallel measures.

21 RELIABILITY 21 There are four basic methods to estimate the reliability of empirical measurements namely the retest method, the alternative-form method, the split-halves method and the internal consistency method The range of values for the reliability method is from 0 to 1. Values near 1 show good reliability. Usually, if the value is more than 0.7 than the method is reliable.

22 RELIABILITY In SPSS, the reliability analysis is obtained from the following commands: ANALYZE SCALE RELIABILITY ANALYSIS Under reliability, there are five different types of methods namely (1) The alpha Cronbach s method 22 (2) The split-half method (3) The Guttman method (4) The parallel method (5) The strict parallel method

23 The easiest method. The Retest Method 23 Suppose that a set of questionnaires or tests are given to some respondents. Then if the same set of questionnaires or tests are given to the same respondents after some specified time period, then this is known as the retest method. The interval between the two tests are usually taken to be from two to four weeks.

24 The Retest Method The equations for the two tests are as follows: Assumptions: (i) (ii) Thus and 24 X 1 X t t X X t V V 1 2, x X X

25 The Retest Method 25 Weaknesses of this method 1. Researches usually cannot do more than one tests 2. The reaction of the respondents about certain surveys. Example: If a respondent is being interviewed about whether he/she will vote in a coming election at time 1, the respondent might make a decision at time 2 and will actually vote at time 3 due to the fact that he/she was sensitized to the election through the interview.

26 The alternative form method 26 The most frequently used method in the field of education Similar to the retest method as it requires two sets of tests which is given to the same respondents Different from the retest method in that an alternative form of the questions are given The two sets of questions must measure the same thing. Example: If two tests are designed to measure the understanding of mathematical operators using 20 questions for each test then the sets of questions must be of equal difficulty

27 The alternative form method Superior than the retest method 27 Weakness of this method is to design questionnaires or tests which are of equal level.

28 The split-half method Suppose there are N questions in a questionnaire These questions are split into two equal halves each having N/2 questions. 28 Split can be done arbitrarily. Example: Can choose the first questions and the other being the last questions or we can choose the even numbered questions for the first half and the odd numbered questions as the second half. The value of the reliability measures will be different for different splits.

29 The split-half method The Spearman-Brown prophecy formula for measuring reliability is given by : xx 2 xx 1 xx 29 xx is the reliability for the whole sample xx is the correlation between the two split-half

30 Internal consistency method 30 This method require only a single test on the sample and is usually known as the internal consistency method. The most popular of the internal consistency method is the Cronbach s alpha method which is given by N [ 1 ( N 1)] N is equal to the number of questions is the mean correlation between each questions

31 CONTENTS Reliability And Validity Sample Size Determination Sampling Designs Descriptive And Inferential Statistics Pilot Study 31

32 Sample Size Determination 32 Population Sample Sample is chosen at random from a population

33 Sample Size Determination Sample size depends on the budget and degree of confidence required. Smaller samples are more likely to be different from the population than larger ones. So smaller samples have more sampling error and lower reliability. Sample Size 33 Sampling error Sample Reliability

34 Sample Size Determination Krejcie, R.V. and Morgan, D.W. (1970), Determining Sample Size For Research Activities, Educational And Psychological Measurement, 30, The following Table 1 is from this paper. 34

35 Sample Size Determination 35 N S N S N S N is the population size; S is sample size

36 CONTENTS Reliability And Validity Sample Size Determination Sampling Designs Descriptive And Inferential Statistics Pilot Study 36

37 SAMPLING Foundation of a good sample survey is the sample. A sample is some part of a larger body specially selected to represent the population. Sampling is the process by which it is done. Samples must be representative of the population. 37

38 PROBABILITY AND NONPROBABILITY SAMPLING 38 Probability sampling is a process of sample selection in which elements are chosen by chance procedures and with known probabilities of selection. Nonprobability sampling includes all methods in which units are not selected by chance procedures or with known probabilities of selection.

39 NONPROBABILITY SAMPLING 39 Haphazard sampling: Samples are made up of individuals casually met or conveniently available such as students enrolled in a class or people passing by on a street corner. Cannot make generalization beyond the collections themselves and are seldom of scientific interest. Also known as convenience sampling. Judgmental or purposive sampling: Sample elements are chosen from the population by interviewers using their own discretion about which informants are typical or representative. Results of such sampling procedure can be very good, if the interviewers intuition or judgment is sound.

40 NONPROBABILITY SAMPLING 40 Quota sampling: Process of selection in which the element are chosen by interviewers using prearranged categories of sample elements to obtain a predetermined number of cases in each category. Expert sampling: Elements are chosen on the basis of informed opinion that they are representative of the population in question. Example: A specialist on secondary education may decide that four schools across the country adequately represent the range of variation seen in teaching methods.

41 PROBABILITY SAMPLING 41 Simple Random Sampling (srs): Each population member has the same probability of appearing in the sample. Sample size: Depends on the objective of survey. Assume that we need to estimate the population mean, by using a srs mean and restricting to an acceptable level the probability that the absolute difference between the population mean and the sample mean is greater than some specified value.

42 Simple Random Sampling 42 Then we have for some given d and α Thus, where

43 Determination of S 2 in SRS 43 From pilot studies From previous surveys From a preliminary sample

44 PROBABILITY SAMPLING 44 Systematic sampling: Method of selecting units from a list through the application of a selection interval, I, so that every I th unit on the list, following a random start, is included in the sample. Sample size: Depends on the objective of survey. Assume that we need to estimate the population mean, by using a sample mean restricting to an acceptable level the probability that the absolute difference between the population mean and the sample mean is greater than some specified value.

45 Systematic Sampling 45 Then we have for some given d and α Thus, where

46 PROBABILITY SAMPLING 46 Stratified (simple) random sampling: Technique where a population can be conveniently partitioned into a set of sub-populations (strata). Such a population is said to be stratified. Within a strata, simple random sampling method is used to determine the sample. Cluster sampling: Sometimes a finite population may consist of a large number of groups of individuals, e.g. of households in a city. This is a special form of stratification (many strata of rather small size) and is referred as clusters. Draw a cluster sample as a srs of the clusters. If all the members of the sampled clusters are obtained, this is known as one-stage cluster sampling.

47 CONTENTS Reliability And Validity Sample Size Determination Sampling Designs Descriptive And Inferential Statistics Pilot Study 47

48 Descriptive and Inferential Statistics Descriptive Statistics- used to describe the data where data are presented in the form of tables, charts or summarization by means of percentiles and standard deviation Measures of locations: mean, median, mode 48 Measures of spread: standard deviation, variance, range. Plots such bar chart, pie chart, histogram, Box and Whiskers plot. Not enough just to do this in your FYP!!!

49 Descriptive and Inferential Statistics 49 Population Sample Sample is chosen at random from a population

50 Descriptive and Inferential Statistics 50 Inferential statistics - Process of drawing information from sampled observations of a population and making conclusions about the population. -Two-prong approach. (1) Sampling must be representative of population (2) Correct conclusions made of population

51 Descriptive and Inferential Statistics 51 Inferential statistics - t-tests - Analysis of variance tests - Chi-square tests - Regression models Students must do some inferential statistics.

52 CONTENTS Reliability And Validity Sample Size Determination Sampling Designs Descriptive And Inferential Statistics Pilot Study 52

53 Pilot Study 53 Last major stage of survey work before data collection stage. Designed to find any problems with the data collection process such as: - Poor introduction and instructions to questionnaire - Unclear or undefined terms - Unclear or ambiguous response task - Too many don t know responses - Biased or offensive questions - and so on. Check reliability

54 Pilot Study 54 Choose potential respondents to complete a questionnaire Other approaches are: - Behaviour coding: Investigator watches the respondent and/or interviewer complete the questionnaire or observes the behaviour after it has been recorded on tape. - Cognitive interview: Respondents are asked to think aloud while completing the survey and to describe everything that comes to mind while arriving at an answer.

55 Pilot Study 55 Interviewer evaluation: Interviewers are asked to code question characteristics and respondent behaviour. Respondent evaluation: Respondents are asked to rate and/or comment about the questions. Expert panels: Experts in survey research can be asked to review a questionnaire and identify potential problems.

56 Guidelines 56 Sample size: At least 25 samples. For behaviour and cognitive interview, sample size can be reduced to about 12 samples. Sample composition: Should be similar to that of the survey. Number of pretests: One pretest is adequate but not always recommended. Data collection time: For interviews, can allow 50% longer than the projected interview. Statistical analysis: Can be done if data is more than 25. Number of identified problems: Will definitely find problems. Measure the reliability of the questionnaire

57 SOME COMMENTS Before the pilot study, the following must be followed: 57 (1) For questionnaires that are adopted and adapted (with less than 20% change), content validity need not be checked. (2) For NEW and adapted (more than 20%) questionnaires, content validity must be done with at least three experts.

58 SOME COMMENTS 58 During the pilot study, for NEW questionnaires, (1)the number of sample size must be more than 100 (2)factor analysis must be carried out.

59 SUMMARY FYP report must include (1) Validity Content Validity (2) Reliability (3) Pilot Study (4) Descriptive Statistics (5) Inferential Statistics 59

60 THANK YOU 60