Dr. Michael Passer Psychology 209 University of Washington 1. Evaluating Measures

Transcription

1 Dr. Michael Passer Psychology 209 University of Washington 1 Evaluating Measures Accuracy: The degree to which an instrument yields results that agree with an accepted standard. Pertains to proper calibration of instruments that measure properties (e.g., time, length, weight, decibels, frequency of occurrence ) of behavior and other events. Examples: A calibrated stopwatch accurately records elapsed time to a thousandth of a second. A human observer -- the measuring instrument in this case -- accurately records the number of times a parent speaks to a child during a 1-hour play session (accuracy judged in relation to the frequency of speaking recorded on a videotape of that session.

2 Dr. Michael Passer Psychology 209 University of Washington 2 Note: For many psychological concepts, the concept of accuracy is not applicable. How can we speak of the accuracy of people s attitudes, creativity, intelligence, SAT scores, degree of depression, and so on? For such psychological measures, we focus on the concepts of reliability and validity (these concepts also apply to physical measures ).

3 Dr. Michael Passer Psychology 209 University of Washington 3 Reliability: extent to which a measure (of the same variable or phenomenon) yields consistent results. Example: You step on a weight scale 3 times within 5 minutes and get the same readout each time. An IQ test is given to 250 children twice within a month, and the scores from the two testing periods correlate very strongly. Validity: extent to which a measure truly assesses what it is claimed to measure; the degree to which a measure achieves the aims for which it was designed. Example 1: A psychological test claimed to measure self-esteem truly does measure self-esteem, and not some other characteristic such as IQ or creativity. Example 2: A job screening test developed to help predict how well applicants will perform on the job actually does help a company select applicants who end up performing well.

4 Dr. Michael Passer Psychology 209 University of Washington 4 Some Types of Validity Face validity: extent to which a measure simply appears to be a reasonable measure of some trait. Scientifically, face validity is the least important type of validity. Example: A test measuring selfesteem "looks valid" to people taking the test because the items seem plausible (e.g., "How often do you feel bad about yourself?")

5 Dr. Michael Passer Psychology 209 University of Washington 5 Content validity: degree to which the content of a measure (e.g., an IQ test, job aptitude test, achievement test) covers a representative sample of the domain (e.g., intelligence, job skills, knowledge) being assessed. Example: A one-hour job simulation test measuring employee's aptitude to be business managers presents them with diverse tasks. These tasks represent the various skills that managers will actually need to display on the job (e.g., verbal, written, social, organizational, and decisionmaking skills).

6 Dr. Michael Passer Psychology 209 University of Washington 6 Criterion validity: degree to which scores on a measure meaningfully predict some current or future behavior (or other criterion; e.g., illness) Example 1: A new standardized test is developed to predict academic performance. 100,000 high school seniors take the test. As a first step, the researcher correlates the test scores with students' current high school gpa and finds a positive correlation. The criterion is current high school gpa. Example 2: A new standardized test is developed to predict academic performance. 100,000 high school seniors take the test. Among those who go on to college, the test scores measured in high school correlate positively with first year college gpa. The criterion is future first year college gpa.

7 Dr. Michael Passer Psychology 209 University of Washington 7 Construct validity: degree to which a construct (e.g., intelligence, self-esteem, aggressiveness) is valid; also, the degree to which a particular instrument is a valid measure of that construct. Two subtypes of construct validity. a) Convergent validity: Scores on a measure should a) relate to scores on other already-validated measures of the same construct, and b) to scores on measures of other constructs to which, according to theory, they should be related. Example 1: We claim that a new test measures "verbal intelligence". Test scores should correlate highly with already-validated verbal intelligence tests. Example 2: We develop a new test to measure shyness. According to theory, shyness and social anxiety are different constructs, but should be somewhat related. We find that scores on our shyness test correlate moderately with scores on an already validated test of social anxiety.

8 Dr. Michael Passer Psychology 209 University of Washington 8 Note: Example 1 above (high correlation between two intelligence tests). Your textbook reports this as an example of "criterion validity." If a researcher's goal is to use a new IQ test to predict scores on another IQ test, then this represents criterion validity. But, the usual reason for correlating a new IQ test with an alreadyvalidated one is to show that the new test "really does measure IQ". This is convergent/construct validity b) Discriminant (Divergent) validity: Scores on a measure should not relate to scores on measures of other constructs to which, according to theory, they should not be related. Example: If, according to theory, shyness and creativity are independent constructs, then scores on a test of shyness should not correlate significantly with scores on an already validated test of creativity.

9 Dr. Michael Passer Psychology 209 University of Washington 9 OTHER POINTS 1. CONTENT AND CRITERION VALIDITY ALSO HELP TO ESTABLISH CONSTRUCT VALIDITY. 2. RELIABILITY IS NECESSARY FOR VALIDITY IF A MEASURE IS NOT RELIABLE, THEN IT CANNOT BE VALID. IN OTHER WORDS, RELIABILITY IS A PREREQUISITE FOR VALIDITY 3. RELIABILITY ALONE IS NOT SUFFICIENT TO ESTABLISH VALIDITY WE CANNOT ASSUME THAT, IF A MEASURE IS RELIABLE, THEN IT AUTOMATICALLY IS VALID. A MEASURE CAN BE RELIABLE BUT INVALID.

10 Dr. Michael Passer Psychology 209 University of Washington 10 Dr. Michael Passer Psychology 209 U. of Washington KNOWLEDGE CHECK QUESTIONS Which of the following concepts is MOST DIRECTLY illustrated by each example? Which other concepts also apply? Low reliability Low face validity Low content validity High reliability High face validity High content validity Low criterion validity High criterion validity Low construct validity High construct validity 1.Weighing yourself on a scale 3 times and getting the following readings: 150 lbs, 157 lbs, 153 lbs 2. Administering a job skills test to 100 prospective employees, hiring the 50 best scorers, and then finding out that even among these 50 new employees, those who scored higher on the job skills test tend to perform better on the job. 3. Students who score in the top 10% on the ACTs (a college aptitude test) tend to score in about the same percentile on the SATs (a different college aptitude test) 4. After many administrations, researchers administering a polygraph test begin to worry that the machine is actually measuring anxiety and not dishonest responses. 5. A personality test that helps to predict the development of schizophrenia consists entirely of items such as What is your favorite color? and Are red apples better than green apples? 6. Individuals that score high on a questionnaire measuring racism on Tuesday morning are likely to score high on the same scale on Tuesday afternoon. 7. Two researchers use a newly developed observational coding system to record how newlywed partners interact with one another. The researchers' goal is to predict which couples will be divorced within 4 years. Results show that the "behavioral profiles" established by the coding system correctly predicted divorce 90% of the time. 8. Research consistently shows that scores on Dr. Smith s "Selfishness Test" are highly positively correlated with scores on other selfishness tests and, as hypothesized, are moderately correlated with scores on tests that measure egocentrism." 9. Suppose shyness and extraversion should be negatively correlated. We find that people with higher scores on a new shyness test (indicating they are more shy) also score higher on a previously validated extraversion test (indicating they are more socially outgoing). 10. Students in Professor Jones' Geography 215 class are assigned to read Chapters 1, 2, 3, and 4 for the first exam. All the chapters are similar in length and amount of material. In lecture Professor Jones conducts 3 lectures on the topics in each chapter. Students are told to study all chapters and lecture notes for their first exam. On the first exam, however, 90% of the exam questions are based on the material in Chapter 3 and Chapter 4, and only 10% of the questions are based on material in Chapter 1 and Chapter 2.

11 Dr. Michael Passer Psychology 209 University of Washington 11 Which of the following concepts is MOST DIRECTLY illustrated by each example? Which other concepts also apply? Low reliability High reliability Low face validity High face validity Low content validity High content validity Low criterion validity High criterion validity Low construct validity High construct validity