Soc320: Reliability and Validity. [Michael Allen] Slide #: 1. Slide Title: WSU Online Title Slide

Transcription

1 Soc320: Reliability and Validity [Michael Allen] Slide #: 1 Slide Title: WSU Online Title Slide Title: Reliability and Validity Speaker: Michael Allen Audio: [Music playing in background] Slide #: 2 Slide Title: Slide 2 Assessing Measures Reliability Can we rely on a measure? Is it subject to random error? Validity Does indicator measure the concept? Does it measure something else? Audio: In this lecture I d like to talk about measurement, particularly issues of how you asses the reliability or validity of the measures you might develop. The terms of reliability and validity have common usage in this language, but social scientists have particular meanings for those terms, and it s those meanings that we want to explore today. The first issue has to do with reliability- what we mean by reliability. Reliability has to do with the extent at which we can rely on a measure. Is it reliable? Another term that we might use is stability? Is it a stable measure? In particular, you want to know whether a measure is subject to random measurement error. Random error is a problem in a lot of social research, and we ll discuss this issue again and again throughout this course. The other issue is validity- how well does an indicator measure the concept that we are interested in, and this is really a tricky problem, one for which there is no easy solution, but we do have some ways of approaching it. So we really want to know when we look at validity whether a measure measures what we re interested in or whether a measure is something else or the indicator is a good measure of a particular concept or not. Slide #: 3 Slide Title: Slide 3 Example: Problems of Poor Measurement Alcohol treatment program Select alcoholics for treatment 1

2 Administer questionnaire to subjects Index of Problem Drinking Use index to assign subjects to treatment High score: Given treatment Low score: Not given treatment Audio: In order to make this whole issue of reliability and validity more meaningful, I want to discuss the problem of formation within the context of an empirical example, and that empirical example is an alcohol treatment program. Now what we want to do in that program is select people for treatment who are in fact alcoholics, people who really do have a drinking problem. We don t want to treat everybody, we don t have enough money to treat everybody, so we want to be very selective in the people that we choose for treatment. Well, one way to do this is to administer a questionnaire to people who might be subjects to this treatment. These could be people who were arrested for drinking under the influence, and we want to select out those people who have been arrested who really have a serious problem, who are alcoholics in some sense and really would benefit from the treatment, and so we might give them a questionnaire asking them a series of questions about their behavior and their attitudes, and this questionnaire could give us an index of problem drinking. We could add up the scores on different items and come up with an index of problem drinking, and we would then use the score on this index to assign some people treatment and to reject others, so the high score means that you have indicated that you have a lot of problems with drinking, and that would therefore qualify you for getting treatment, and other people who might be arrested for drinking under the influence might have a low score, which means they don t have a persistent or chronic drinking problem, and therefore they wouldn t be given treatment. Slide #: 4 Slide Title: Slide 4 Problems Created by Lack of Reliability Random error in scores A few social drinkers score high They do not need treatment They receive inappropriate treatment A few problem drinkers score low They need treatment They do not get any treatment Audio: Let s first look at the problems that would be created by having a major lack of reliability, and that means that we have random error in the scores on our index because of perhaps ambiguity in the items, the way the items are worded, or perhaps even the testing situation. A few people who are only social drinkers, that is people who just drink in social situations and usually not to excess- people who are not really problem drinkers or alcoholicsmight score higher in this index in that given situation on that given day. These people probably really don t need treatment or at least they don t need it as much as other people might, and so 2

3 they would receive inappropriate treatment, and in a sense we re probably wasting money by treating these people who have relatively minor problems. Because of the same random error, problem drinkers might sort of by chance grow low on this index of problem drinking. They probably really need treatment, but because of the situation, because of the questions, they end up with a low score, so they don t get any treatment. In this case, we re not going to have much impact on the problem of alcoholism because a lot of people who deserve treatment aren t getting it. Slide #: 5 Slide Title: Slide 5 Problems Created by Lack of Validity Scores do not measure concept well Measures more than problem drinking Includes items about social drinking I enjoy having a drink at parties Social drinkers assigned to treatment This treatment is not appropriate Audio: In the same situation, there are also problems associated with a lack of validity. In this case, the scores that we developed for each person on this index of problem drinking do not make it the concept of problem drinking or alcoholism very well. They might measure something else. They might measure social drinking, so you might have a question on your index that says, I enjoy having a drink at parties, which is true for a lot of people, but a lot of these people who would agree with that question are not necessarily alcoholics. There are different levels of drinking, and so by including these items on our index of problem drinking, we have a lot of social drinkers who might score high because of questions like that, that are assigned for treatment when this treatment is not necessarily appropriate, so this is the problem we get into with lack of validity. Slide #: 6 Slide Title: Slide 6 Assessing Reliability Assess randomness of measure Reliable measures should be stable IQ scores should change only slowly Test-retest reliability Test: Measurement of variable Measure variable at two points in time Expect strong association between tests Test score should predict retest score Audio: So reliability and validity are problems, and one of the first things we have to do as a social scientist is sort of assess the extent at which these problems might affect our research. Particularly, we can assess reliability relatively easily. We can assess the randomness of the scores that we get using a particular measure or index. Reliable measures are those that should be 3

4 stable over time- think of a case of IQ tests, intelligence quotient, intelligence, scholastic aptitude. Whatever you want to call it, that basic ability doesn t change very much, and if it does change, it changes rather slowly over time, so IQ scores should be relatively stable from one period to the next, particularly if that time period is fairly short. The way we assess the reliability of an IQ test or scholastic aptitude test is through what we call test-retest reliability. Basically we give tests to people at two points in time. We give them the first test, we get a score on it, the scholastic aptitude test score, and then we come back and we use that exact same questionnaire, those same questions again, at a later point in time to see where they get the same score. We expect that we have very strong associations between the test and the retest. The test score should predict retest score, and if that s the case, then we have reliability. If that is not the case, then we have a measure that s unreliable. Slide #: 7 Slide Title: Slide 7 Contingency Table: Test-Retest association Problem Drinker: Similar results over time Time 1 Yes No Time Yes No Audio: We haven t talked in this course about statistical analysis or even something as simple as contingency table, but we can think about test reliability in terms of the association between a test and a retest. We can give someone a test of index of problem drinking, a series of questions, and it should give similar results over time if in fact that index is reliable measure of problem drinking, and you can see this in a table. If we gave a test to 100 people in time 1 and the same people- these are the very same people now- the very same test again a few weeks later, we would expect their scores to be the same or at least very similar from one test to the next, so if a test predicts that someone is a problem drinker at time 1, it should predict that they re a problem drinker at time 2, and if they re not a problem drinker at time 1, they should not be a problem drinker at time 2. There might be a little slippage from one test to the next because people might be on a borderline between being a problem drinker on one test and not being a problem drinker on the next, but by and large, you should be able to make a prediction of test score at time 2 from the test score at time 1. Slide #: 8 Slide Title: Slide 8 Problems of Test-Retest Method Selecting proper time period Short time period: Memory effects 4

5 Subjects try to give same answers Effect: Overestimate reliability Long time period: Change in subjects Variable may change over time Effect: Underestimate reliability Audio: The test-retest method for assessing reliability [inaudible] and index for measure over time is the very best method that we have, but there are problems associated with it that we should be aware of. Slide #: 9 Slide Title: Slide 9 Example of Test-Retest Reliability SAT scores: Must be reliable May determine admission to college Give same test to same student twice Enough time between tests Students do not remember questions Not too much time between tests Change in scholastic aptitude Ideal time: Few months Audio: The first problem is that we have to select a proper time period. One example of the issue of estimating test-retest reliability and how time might factor into that has to do with SAT scores. SAT scores, scholastic aptitude tests, are very important. They must be reliable. They determine whether or not you re allowed to enter college or not or what college you re allowed to attend, so they re very important tests, and they should be very reliable as well as a very valid measure for your ability. Well the way that we estimate the reliability of these scholastic aptitude tests is to give the same test to the same students twice. It s important therefore that we allow enough time between the tests so that the students simply are not recalling their answers to the questions on the first test on the answer on the second test, so that s the first thing we have to sort of eliminate, the sort of memory effects. At the same time, we don t want too much time to pass between the test and the retest because the basic underlying concept, the basic thing we re trying to measure, which is scholastic aptitude, can change over time. People can become better at doing scholastic work; other people can sort of decline in their ability to do scholastic work, so we have to choose the ideal period when people no longer remember their responses to the first test, but they have not sort of changed their fundamental attitudes and behaviors of whatever for the second test. So the ideal time in the case of scholastic aptitude test in terms of assessing the test-retest reliabilities are probably a few months. Slide #: 10 Slide Title: Slide 10 Reliability of Index Sum of scores on several indicators Random error might affect each indicator 5

6 Idiosyncratic variation Index is more reliable than any one indicator Errors cancel one another out Positive error can offset negative error The average is the sum divided by number of items Average score is more reliable than single scores Audio: In the context of reliability, I want to say a few words about the reliability of an index. An index is a score. It is derived by adding up the scores on a series of indicators, individual indicators. In the case of a scholastic aptitude test, for example, you get a certain number of points for answering each question correctly, and all of those points are added up to give you an overall score on the scholastic aptitude test. Well I want to mention that we trust indexes more than we do single items for a number of reasons. Each individual question that we ask, whether it s an aptitude question or attitudinal question or behavioral question, they are subject to a certain amount of idiosyncratic variation. In the case of school aptitude, you might have missed lectures on a certain topic, so you would not be able to answer questions on that topic very well, but otherwise you might be a very good student, so we wanted to eliminate the random error that is associated with each individual indicator and sort of get an overall measure of your ability. In general, an index is more valid than any one indicator. The basic idea is that errors cancel one another out. Positive error accidentally getting questioned right can be offset by another question that they accidentally got wrong, so the index sort of cancels out those positive and negative errors. The average therefore is more reliable than individual scores, and of course the average is simply the sum of all the scores on the individual items divided by the number of items in the index. Slide #: 11 Slide Title: Slide 11 Assessing Validity Internal validity: Multiple indicators Indicators should be associated with one another They measure the same concept Item analysis: Select best indicators Those with strongest associations Drop indicators with weak associations They may measure different concept Index: Combination of indicators Audio: We also have procedures for assessing the validity of a measure, particularly if that measure is an index, and one form of validity is what we call internal validity, and think of a situation we have multiple indicators that are all presumed to be measures of a particular underlying abstract concept. Well if they re all indicators of the same concept, they should be related to each other, they should be associated with each other. If you can do well in Calculus, you should do well in Algebra, you should do well in other sort of mathematical skills, so those questions, those indicators of mathematical ability, would be related to each other. One way of looking at this or one way of determining the validity of individual indicators is what 6

7 we call item analysis, and item analysis we look at the associations between different indicators, and if we find an indicator that is not associated with the other indicators, we have to assume that it measures something else, and so we take that indictor out of the index, we drop it from the analysis, and we simply use those indicators that have a high association with one another. We ll see this in attitudinal research where we re very vague about what constitutes a particular attitude, but if we have several indicators that seem to be associated with each other, several questions that people respond to in a consistent way, then we assume that they are valid measures of that underlying concept, and that s one reason why we prefer indexes over individual indicators because, once again, they contain a number of items that all presumably measure the same thing. Slide #: 12 Slide Title: Slide 12 Example of Internal Validity Concept: Alienation from work Indicators: Attitudinal questions I do not take any pride in my work I find the work I do meaningless I hate my job Indicators of same concept Alienated workers: Agree with all three Non-alienated workers: Disagree with all three Audio: Let me give you an example of how we might assess or at least conceptualize internal validity. One of the sort of underlying concepts in sociology of work is the notion alienation. Some people just simply do not like the work they do, and as a result, they re not very good workers, so we might ask a series of questions that try to get at the fact that some people are more alienated than others. These might be more attitudinal questions, so we could ask, for example, three questions: The first might be, I do not take any pride in my work. If you agree with that then you re probably an alienated worker. The second question is, I find the work I do meaningless. Well once again, if you re an alienated worker, you re going to probably say yes to that question. And finally, straight up, I hate my job. Well, alienated workers are going to agree with that statement. These are all indicators of the same concept, and if you are a truly alienated worker, if you really dislike your job and don t like the work you re doing, you re going to agree with all three of these questions. On the other hand, if you like your work, and there are people who like their jobs and like the work they re doing, you can disagree with all three of these times, so we can take these three items and put them in an index. Give you a score of 1 for each one that you agree with, and if you have a very high score, a score of 3, meaning you agreed with all three of these statements, then you are probably a very alienated worker. If you get a score of 0 on this index, then that means you disagreed to all three questions, and you are a non-alienated worker. Slide #: 13 Slide Title: Slide 13 Assessing Validity 7

8 External validity: Indicators or index Requires external Criterion Index should predict other variables Attitude should predict behavior IQ scores should predict academic achievement Early IQ test: No external validity Revision of test: More verbal items SAT scores predict college grades Audio: There are other equally important ways of assessing the validity of a measure or index, and one of those is external validity. We want to look at either indicators or indexes and see whether or not they predict some external criterion, some behavior or attitude that they should predict, theoretically, that are not necessarily the same concept. An index should predict other variables; attitude toward work, for example, should predict behavior. We can see this in particular with IQ test scores or SAT scores. If they are valid measures of your scholastic aptitude, your intelligence, then they should predict academic achievement, and that is precisely the problem with the early IQ tests, the Stanford Binet test that was given at Stanford Universityit had no external validity that was given to both men and women, and it turned out that women did not do as well as men on that test, but the paradox was that women did better in school than men, and when they looked at that test, they realized that the test had a bias in it. It had more mathematical questions than it did verbal questions, and so they revised the test so that it had equal number of verbal and quantitative questions, and sure enough women scored equally as well as men, and it becomes a much better predicator of how people do in school. SAT scores are used primarily to predict how well you do in college. They are used as a predictor of your freshman GPA. If they couldn t predict freshman GPA, they would not be of much use. 8