PSY Test Validity

Transcription

1 PSY 4050 Test Validity Now that we know how to determine if a test is reliable or consistent, we need to know if it is reliably good or reliably bad. My business sense is reliably poor at making me money. If a test is reliably poor, it is not of much value in assessing behavior. The term Validity refers to the capacity of a test to assess what it is intended to assess. If an intellectual test is valid, it will adeptly assess one's intelligence (whatever that is). The book's definition of validity is...the agreement between a test score or measure and the quality it is believed to measure. More importantly, validity has to do with the correctness of inferences drawn from the score of any particular test (i.e., GRE is supposed to predict how well one does in graduate school...does it? the answer, yes...a little) Therefore, are inferences about how a person might do in grad school based on the GRE valid?...a little. In test development one is to assume that a test is not valid until it is proven to be. That evidence comes from showing a relationship between the test and other variables (what type of relationship???? correlational or regression). Compare the results of your measure to some external standard e.g. if I made a new ruler from scrap wood at home is it a valid ruler? I would take it to the Bureau of Weights and Measures and compare it. If it matches, it is a valid measuring instrument, if not, then inferences about length I made based on my ruler would likely be inaccurate. The trouble with psychological measures is that there are few "external standards" such as the Bureau of Weights and Measures to compare our measures to. Psychologists must collect evidence from a number of other sources to determine validity of their measures. There are different ways to check validity based on different types of validity that exist Face Content Criterion Face Validity- the mere appearance that a measure has validity. If what you observe is exactly what you are interested in face validity is sufficient. if e.g. If I am interested in how quickly an infant learns to drink out of a cup, I simply watch and record how long it takes. This is face valid, however, I am more interested in extrapolating an infant's intellect by how fast that

2 infant learns to drink from a cup, this may not be a valid measure, even though it is face valid. Another issue with face validity is if respondents feel the measure is face valid. If a person taking a reading comprehension test is presented with a paragraph and is instructed to read the paragraph, remember all possible, and be able to answer questions later, this is face valid. On the other hand, if a person taking the same test is given another test that measures concentration, but has nothing to do with reading (seemingly) the person is less likely to perform to their optimal ability...they see the test as invalid, even if it is a valid part of reading comprehension. cover. Content-Related Validity-a test has content validity to the extent that it provides an adequate representation of the conceptual domain it is designed to covered astrophysical e.g. do my exams adequately represent your learning of the material in the book or in class, or do I test on some third dimensional level? Evidence is logical rather than statistical. Examine the test and see if it makes logical sense. Does a test with 10 math questions seem a valid measure of verbal reasoning? Not hardly. Answers the question.."has the test been constructed properly?" With just content validity, you must consider all factors that could have influenced the results of the test to make an accurate generalization about what the test score really means. e.g. illness, fatigue, poor lighting conditions, forgot my glasses, etc. This type of validity has several shortcomings e.g. a test of management skills may show someone is very knowledgeable at these types of skills, but personality-wise they may not possess the interpersonal abilities to be an effective manager. There are three steps to determining content validity 1. Describe the content domain To assess content validity you need to know something about the Domain you are measuring, that is the structure and the boundaries that entail the thing you are interested in e.g. if you are interested in assessing management style, then you need to know something about management and the things that make a manager good.

3 It is likely that the thing you are measuring is made up of several sub-domains. Knowledge of the specifics of these sub-domains is critical e.g. good management style often includes: personnel decisions program planning interpersonal skills delegation abilities 2. Determine the areas of the content domain that are measured by each test item do all of the items on the test refer to some component of good management style? Which items belong in which domain or sub-domain If the majority of items are outside the test domain, it would be difficult to have good content validity 3. Compare the structure of the test with the structure of the content domain Does the test cover all of the domain, only a specific part, or not at all? Tests are used to help make decisions about people or things. The accuracy of decisions made based on a test is directly linked to the validity of a test score. An invalid test could lead to erroneous decisions. The simplest method of determining whether a test can be used validly in making decisions is to correlate test scores with measures of success or of the outcomes of decisions (assess GRE predictive of graduate work by looking at GRE and Graduate GPA or completion of program). These measures are called criteria and are often used in determining the validity of a test. Choosing the criterion carefully and defining them specifically are the only ways to know if a measure is really valid. Criterion Related Validity- refers to the relationship a test has with a certain, predefined criterion or minimum expectation. How well a test predicts behavior in the future, or how is related with another measure are two different types. Predictive Validity-The quality of forecasts made by a test e.g. the SAT predicts how well a student will do in college e.g. The ASVAB (Armed Services Vocational Aptitude Battery) predicts which job placement a person would be most successful in. It predicted success in training for the occupation but had not been tried with job success. The Army undertook a study to determine if the ASVAB predicted job performance, but the definition of good job performance (the criteria) was poorly defined and the study flopped. Should the test be done away with? Or should the criteria be adjusted?.

4 How to test Predictive Validity 1- Obtain test scores from a group of applicants, but do not use these test scores to determine hiring rank. 2- At some time later, obtain performance measures for those persons who took the test and correlate their test scores with their work performance measures. But who wants to hire a bunch of clowns if you have test scores that might make some difference in who would be best and who will fail. Doesn't make much financial sense. The practical solution is to obtain both test scores and criterion scores at the same time or concurrently. the Concurrent Validity-examines the simultaneous relationship between the test and criterion. performance? How well does a test of learning disability related with school that Industrial psychologists who are involved in employee selection found taking job samples that are related to the job they are hiring for yielded good insight in to how a perspective employee might perform on the job. This relationship is usually expressed as a validity coefficient derived through correlation analysis. The validity coefficient squared is the percentage of variability accounted for by the predictor. If the SAT has a validity coefficient of.40 predicting performance in college, for a validity coefficient of.40 we know that 16% of the variability is explained by the test. Thus, 16% of a person's success in college is accurately predicted by the SAT. The remaining 84% is a mystery. Many factors can affect a test's criterion validity and make it seem more valid than it really is. Changes in the causes of the relationship e.g. a strong relationship between a test of political prowess and immoral values may surface now in light of recent events. Meaning of the criterion e.g. what exactly does the SAT predict? Information learned, success on a personal level, or gpa...gpa and what does that mean? Subject population of the validity subject group. Was the test developed with people for whom the test was intended to be used. Most tests are developed with young middle class, single, white, adults (the freshman population). Are these the people for whom inferences are to be made based on test scores?

5 Sample Size-the sample size must be adequate in order to draw any kind of inference back to a group of people. IF the sample size is small, like most are, is there a cross validation where the test is given to an independent sample of people with similar results? Check for restricted ranges that make validity more difficult (GRE and GPA not highly correlated due to restricted range of graduate students...mostly high GPA) Often times a good "criterion" is hard to find. There are few "gold standards" against which to compare new tests or measures. If there are gold standards, then what do we need a test for? The test is needed if it is more economical to measure than is the gold standard. Other times, psychology works with abstract terms called constructs, for which there is no specified criterion. As such, a different type of validity is needed for tests that deal with these abstractions or constructs. Construct Validity -refers to how accurately a test measures a particular psychological construct(s) (which are abstract summaries of some regularity in nature, i.e., self esteem, self efficacy, intrinsic motivation, etc.). poor Constructs are usually impossible to see, but their influence is readily observable i.e., poor self image cannot be seen but behaviors that seem to indicate self worth are abundant To test accurately, the abstract construct must be translated into concrete terms. How accurately these terms reflect the hypothetical idea is construct validity. What behaviors define this construct and which behaviors are not consistent with this construct. Construct validity involves assembling evidence about what a test really means and is done by showing a relationship between the test and other tests and measures. Each time a relationship is found, a piece of the construct is attached to the test. It is an ongoing process. Construct validity is determined by two types of evidence; convergent and discriminant. the Convergent Evidence-found in two ways: 1. when a test correlates with other tests believed to measure the same construct. 2. when specific relationships are demonstrated that could be expected if test is really doing its job. Does the tests provide support or evidence of theoretical links that should be there. e.g., does a measure of depression pick up on all different symptoms of depression? e.g., does a measure of child abuse correlate with those constructs found to be associated with child abuse?

6 Discriminant Evidence-does the test add something to knowledge in the field that is not readily available without it? Does it tap some different sort of construct or combination of constructs?