How Should Validity Testing Impact Expert s Opinion?

Transcription

1 Archives of Forensic Psychology 2015, Vol. 1, No. 2, c 2015 Global Institute of Forensic Psychology ISSN How Should Validity Testing Impact Expert s Opinion? Joseph Lockhart Department of State Hospitals, Sacramento, California 1305 North H Street, #117, Lompoc, CA josephjlockhart@ gmail.com Phone: Symptom validity testing (SVT) and performance validity testing (PVT) have become integral tools in forensic evaluation, and they are formally recommended by both researchers and the major associations of American neuropsychologists (e.g., Bush et al., 2005; Heilbronner et al., 2009; Rogers, 2008). Despite the creation of malingering detection systems (e.g., Bianchini, Greve, & Glynn, 2005; Slick, Sherman, & Iverson, 1999), there is little reliable and valid direction as to how SVTs and PVTs might be used in combination with data derived from other assessment sources. Given the increasing use of SVTs and PVTs, experts could benefit from guidance on how examinees failures in validity testing may be utilized both in developing their assessment hypotheses and in defending their final opinions. This paper discusses how pertinent ethical guidelines, combined with legal elements of admissibility, might be used by forensic psychologists to develop or modify their opinions. Although SVT and PVT testing plays an essential role in assessing effort, this article highlights the fact that, ultimately, it is the forensic experts, not the validity tests or detection models, who make any attributions of exaggeration or malingering. It is also experts who must defend their opinions, whether in a report, during a deposition, or under cross-examination. Keywords: Malingering, Feigning, Symptom Validity Testing, Expert opinion In order to arrive at sound expert conclusions, forensic psychologists seek reliable and valid information. Experts typically gather evidence from three broad informational domains: (a) the examinee s self-report (including the clinical interview and symptom questionnaires), (b) psychological testing, and (c) collateral information (Kane, 2006). This information can be marred if examinees responses do not accurately portray their symptoms and limitations. Examinee exaggeration or feigning is the major cause of bias in disability and personal injury assessment (e.g., Green, Rohling, Lees-Haley, & III, 2001). The evaluation of distortion or bias is essential in the forensic evaluation, because of the mandate that forensic practitioners strive to be unbiased and impartial, and avoid partisan presentation of unrepresentative, incomplete, or inaccurate evidence that might mislead finders of fact (APA, 2013, p. 9). 78

2 impact of validity testing on expert opinion SVTs and PVTs are a primary safeguard against such bias in psychological evaluations. Broadly speaking, SVTs and PVTs can be categorized as indicators of either cognitive or psychiatric response bias within their respective symptom domains (Demakis, Gervais, & Rohling, 2008; Zakzanis, Gammada, & Jeffay, 2012). PVTs are typically associated with measures of attention, cognition, and memory; SVTs are broadly associated with psychiatric symptoms, such as depression, anxiety, or psychosis. PVTs are formally recommended for routine use by the major associations of American neuropsychologists (i.e., National Academy of Neuropsychologists [NAN] Bush et al., 2005; American Academy of Clinical Neuropsychology [AACN], Heilbronner et al., 2009). Rogers (2008) recommended that when the outcome of an evaluation has important consequences, malingering should be systematically evaluated. Its neglect is a serious omission (p. 7). The current article first provides a brief overview of the importance of PVT testing in cognitive assessment, and SVT testing in personality and emotional assessment. SVT and PVT failure is associated with large effects on important psychological measures, particularly neuropsychological and psychological testing and the examinee s self-report, thus directly impacting the quality and usefulness of two of Kane s (2006) three informational domains (the third being collateral information). Next, the article discusses important limitations in the use of SVTs and PVTs. These limitations include their inability to assess intent, variability in classification utility in different clinical populations, and potential for increased false-positives when using multiple tests. The concluding section discusses how forensic psychologists might use these empirical findings, combined with appropriate legal and ethical guidelines, to develop their opinions. To minimize potential miscommunication and misunderstanding, the word malingering is used minimally in this paper; the more neutral terms feigning or symptom exaggeration are employed instead. The term malingering carries highly pejorative and negative connotations. By itself, SVT does not determine whether examinees are malingering, only whether their behavior is consistent with malingering (Iverson, 2006). When working within a forensic context, the role of experts is to assist triers-of-fact in reaching their opinion; a malingering attribution involves judgments of intent that are typically beyond the available data (e.g., Rogers, Bender, & Johnson, 2011). It is important to note that the current review is not exhaustive, but is primarily focused on the use of SVTs and PVTs in the evaluation of exaggerated neuropsychological and psychological injury in civil forensic contexts. Implications of PVT or SVT Failure Malingering, or feigning, is a serious threat to the validity of forensic evaluation. In neuropsychological assessment, several studies (e.g., Green et al., 2001; J. E. Meyers, Volbrecht, Axelrod, & Reinsch-Boothby, 2011; Stevens et al., 2014) have shown that poor effort (as measured by PVT failure) accounts for more covariance in neuropsychological testing results than age, education, or even severity of neuropsychological injury. In objective personality assessment, SVT failure is associated with significant increases on clinical scale elevations (e.g., Wiggins, Wygant, Hoelzle, & Gervais, 2012). This occurs on commonly used tests, such as the Minnesota Multiphasic Personality Inventory (MMPI-2; Butcher, Graham, Ben-Porath, Tellegen, & Dahlstrom, 79

3 lockhart 2003) or the Personality Assessment Inventory (PAI; Morey & Boggs, 1991). Subjects failing SVTs or PVTs often perform worse than subjects with well-documented injuries or disorders (Silk-Eglit, Stenclik, Miele, Lynch, & McCaffrey, 2013). Along with exaggerated responding on objective neuropsychological and personality measures, examinees failing SVTs and PVTs appear to exaggerate their self-reported symptoms (e.g., Armistead-Jehle, Gervais, & Green, 2012; Forbey, Lee, Ben-Porath, Arbisi, & Gartland, 2013). In addition, PVT failure is associated with idealized (supranormal) self-reports of preinjury functioning (e.g., Gunstad & Suhr, 2001, 2004). Forensically speaking, these findings have a direct impact on the major sources of information used by experts (i.e., psychological testing) in their process of causal reasoning. As discussed later in this article, the distortions caused by feigning make this data unreliable from empirical, ethical, and legal perspectives; experts cannot reasonably rely upon this information to form their opinion (e.g., Federal Rules of Evidence 702 (FRE 702); EPPCC; SGFPA). Neuropsychological Test Performance As previously stated, PVTs are typically associated with measures of attention, cognition, and memory. Some commonly used cognitive PVTs include the Test of Memory Malingering (TOMM; Tombaugh & Tombaugh, 1996), Word Memory Test (WMT; Green, Allen, & Astner, 1997), Validity Indicator Profile (VIP; Frederick, (1997)), and Victoria Symptom Validity Test (VSVT; Slick, Hopp, Strauss, & Thompson, 1997). The association between cognitive PVTs and cognitive or neuropsychological test performance has been extensively studied (e.g., Carone & Bush, 2012; Rohling, 2002; Sollman & Berry, 2011; but see McGrath, Mitchell, Kim, and Hough 2010 for an alternate perspective). Failure on cognitive PVTs is associated with poorer performance on nearly all cognitive and neuropsychological measures studied (e.g., Demakis et al., 2008; Green et al., 2001). These include measures of memory, attention, executive functioning, and reaction time. The magnitude of the difference is striking; subjects failing PVTs often perform worse than examinees with well-documented cognitive or psychological deficits. In a recent illustrative article, J. E. Meyers et al. (2011) studied a sample of 314 clinical and forensic referrals with mild traumatic brain injury (mtbi) using the Meyers Neuropsychological Battery (MNB; J. Meyers, 2004). The correlation between the number of PVTs failed and the overall test battery mean (OTBM) was , suggesting that a majority of the covariance in performance could be accounted for by failures on PVTs. Increased Reports of Psychological Symptoms Examples of psychological SVTs typically include validity scales from objective personality tests (e.g., F, FBS, RBS scales from the MMPI-2; Fp from the MMPI-2-RF; NIM scale from the PAI), as well as structured interviews (e.g., M-FAST; Miller, 2001) and SIRS (Rogers, Gillis, Dickens, & Bagby, 1991). Consistent with findings in cognitive testing and PVTs, failure on psychological SVTs is associated with increased endorsement of symptoms on objective psychological tests (e.g., MMPI-2 clinical scales). These scale elevations are typically large and robust: In one of the 80

4 impact of validity testing on expert opinion largest studies conducted by Wiggins et al. (2012), more than 2,200 subjects who had been evaluated for disability were divided into overreporting (approximately 25%) and non-overreporting groups, based on MMPI-2-RF validity scales. Subjects who failed SVTs (compared to those who passed) demonstrated approximately a 1 to 2 SD elevation on all the MMPI restructured clinical (RC) scales. Correlations were attenuated between the MMPI RC scales and external symptom measures in those who failed SVTs; the MMPI scales were essentially measuring SVT performance, not clinical symptoms. Increased Reports of Subjective Impairments Failure on cognitive SVT/PVT tests is closely linked to self-reported complaints of memory and cognitive impairment, but not with actual neuropsychological deficits (e.g., Armistead-Jehle, Gervais, & Green, 2012). On psychological scales (e.g., the MMPI), SVT/PVT failure is associated with large and significant increases in self-reported subjective clinical symptoms (e.g., on the BDI, BAI, and DAPS; Forbey et al., 2013). Such findings of exaggerated self-report are of particular concern to the forensic evaluator because these measures are commonly used to assess the examinee s current level of symptoms and impairment. They are also used to corroborate the information gleaned from the clinical interview. Indeed, Kane (2006, p. 26) referred to these measures as information-gathering instruments because they are typically face-valid and transparent. Armistead-Jehle, Gervais, and Green 2011 correlated performance on a PVT (i.e., the Medical Symptom Validity Test, MSVT; Green & Allen, 2004) with subjective memory complaints (using the MCI) in a forensic disability sample. Although the overall correlation between MSVT and MCI scores was significant (0.44), these results did not appear to reflect actual neuropsychological impairment. When examinees who failed PVTs were excluded from the analysis, there was no significant correlation between subjective self-report of memory complaints and performance on objective cognitive tests. In a similar study, Iverson, Lange, Brooks, and Lynn Ashton Rennison (2010) Iverson, Lange, Brooks, and Rennison (2010) studied mtbi claimants on a measure of postconcussive symptoms (the British Columbia Post-Concussion Symptom Inventory [BC-PSI]; Iverson & Lange, 2003). Subjects who failed a cognitive PVT (the TOMM) scored an average of 1 SD worse on the BC-PSI than those who passed the PVT. Increased Self-Reported Psychological Complaints. Similar to the relation between cognitive PVTs and subjective memory complaints, failure on psychological SVTs is associated with significantly higher scores of self-reported symptomatology on common anxiety, depression, and trauma scales (e.g., the BDI, DAPS). In a large study of more than 1,000 subjects from a wide variety of settings (correctional, psychiatric, and academic), Forbey et al. (2013) studied the relationship between SVT failure on the MMPI-2 and several external measures. These included the BDI and the Perceptual Aberration Scale (PAS; Chapman, Chapman, & Raulin, 1978), among others. For those examinees failing validity scales, the median effect size across all external measures was approximately 1 SD. Demakis et al. (2008) studied 301 claimants evaluated in a forensic context with symptoms of PTSD. The researchers administered multiple depression and anxiety 81

5 lockhart measures that included the BDI, BAI (Beck, Epstein, Brown, & Steer, 1988), and DAPS; cognitive measures including the WAIS (Wechsler, 2008), WRAT (Wilkinson & Robertson, 2006), and WCST (Grant & Berg, 1993); and psychological SVTs (e.g., FBS, Fp, DAPS Negative Bias scale). Demakis et al. found that composite psychological symptom scores increased by roughly 1 SD with each additional failed SVT. In a similar study, Rohling (2002) found that SVT failure was associated with large increases in self-reported depression (e.g., an increase in mean BDI scores from 15.8 to 25.2). Overly Positive Reports of Preinjury Functioning Forensic evaluators frequently need to estimate preinjury abilities and compare them with current functioning. Accurate evaluation of this difference is essential in personal injury and many disability cases. As Kane (2006) stated, The issue is not, per se, how damaged the individual might now be, but, rather, how different the individual is from the way he or she was prior to the traumatic incident (p. 36). The Good Old Days bias, a term coined by Gunstad and Suhr (2001, 2004), which refers to the tendency to positively distort preinjury functioning relative to present (postinjury) functioning, thereby magnifying the apparent effect of the injury or damage. Examinees recall of their preinjury functioning and subsequent symptoms is often imperfect or biased for reasons not associated with feigning. These include examinees expectations regarding their illness as well as the utility of their illness as an explanation for their symptoms (i.e., Ferguson, Mittenberg, Barone, & Schneider, 1999). Other common distortions that affect self-report include telescoping, which involves combining remote past illness events while overemphasizing recent events (e.g., Barsky, 2002), and saliency or availability effects, which involves placing greater emphasis on more memorable illness events. The Good Old Days Bias Research by Gunstad and Suhr (2001, 2004) suggested that it is common for individuals who have suffered significant illnesses and injuries (e.g., headache, head injury, pain) to exaggerate their preinjury functioning and minimize prior problems (i.e., as supranormal ) compared to healthy, uninjured controls. Gunstad and Suhr termed this the good old days bias. Although this effect appears to be common among individuals who have experienced an injury or deficit, it is particularly pronounced in those examinees who fail PVTs. Iverson et al. (2010) studied examinees with mtbi referred to a concussion clinic. They compared mtbi patients who passed PVTs with those who failed PVTs, along with healthy controls. Individuals failing PVTs reported fewer preinjury symptoms than either healthy controls or credible subjects with documented injuries. In practice, the good old days bias suggests that the forensic evaluator must be cautious in accepting the examinee s uncorroborated reports of his or her preinjury functioning. This is especially true for examinees who have failed PVTs and are thus particularly likely to minimize their prior problems as well as exaggerate their present changes in function and impairment. 82

6 Section Summary impact of validity testing on expert opinion Failure on SVTs and PVTs is associated with forensically significant impairments in performance and exaggerated symptom reporting. Such findings of exaggerated self-report are of particular concern to the forensic evaluator because these measures are commonly used to assess the examinee s current level of symptoms and impairment. This tendency to overemphasize symptoms is apparent not only for objective cognitive or personality tests, but also for examinees subjective self-report of symptoms. In addition, PVT failure is associated with idealized (supranormal) self-reports of preinjury functioning. There are broad implications for forensic assessment: When examinees fail SVTs and/or PVTs, it reasonably suggests significant levels of negative distortion across critical sources of information. This includes examinees objective testing results as well as their self-reports about both current and preinjury symptoms, abilities, and functioning. Malingering Detection Models Tests and Limitations As with all psychological tests, SVTs and PVTs have limitations based on their validity and applicability. They cannot transcend the inherent limitations of their classification accuracy (e.g., Bigler, 2012; Mossman, 2000). Some specific limitations of SVTs and PVTs include their inability to assess intent; the potential unreliability of collateral information (making classification of external incentives difficult); the interpretation of failure to a particular cognitive or clinical domain; and reduced specificity in some clinical populations (as discussed later in this paper). A thorough assessment of symptom exaggeration involves a much more complex analysis than whether the examinee passes or fails SVTs or PVTs. Models of Malingering Detection The well-known models of malingering detection/classification created by Slick et al. (1999, modified in 2012) and Bianchini et al. (2005) provide a method to organize data collected during assessments when making a determination of malingering or related negative response bias. The models share several characteristics. As their first criterion, both models require an external incentive as the threshold criterion for a diagnosis of malingering. These models also require the existence of additional evidence that is either inconsistent with known disease entities or suggestive of feigning (e.g., PVT failure). Finally, both models require that the symptom reports and testing results not be better accounted for by actual disorders than by feigning. Limitations of SVT and PVT Interpretation Inability to assess intent.. Though the Slick et al. and Bianchini et al. models are widely accepted, empirical validation of these systems is limited (e.g., Bender & Matusewicz, 2013; Rogers et al., 2011). The most salient criticism is that both external incentives and individual motivation (essential elements of the malingering attribution) are psychological constructs, which are neither well defined nor easily measured. 83

7 lockhart A key limitation of SVTs and PVTs is that they are unable to assess intent, even though they are frequently associated with that label in the forensic context (DeClue, 2002). As Rogers (2008) noted, intent is the defining distinction between malingering and feigning. Rather than a focus on intent, Rogers suggested that feigning or malingering is best explained through an adaptational model. In Rogers model, symptoms are either exaggerated or minimized based upon the perceived demands of the situation, particularly when other options are limited. Limitation of interpretation to SVT and PVT domains. There is little support for an overarching or general factor underlying SVT or PVT failure across domains (e.g., depression, inattention, or even deceptiveness; Lilienfeld, Thames, & Watts, 2013). Consistent with an adaptational/situational model, failure on either neuropsychological (cognitive) or objective personality (psychological) SVTs and PVTs is primarily associated with increased impairment ratings within that respective domain. Therefore, failure in one domain cannot be assumed to predict symptom exaggeration in another domain (Nelson, Sweet, Berry, Bryant, & Granacher, 2007; Zakzanis et al., 2012). Support for the domain-specific nature of SVT and PVT validity comes from several sources, including factor analysis and both convergent and divergent validity (e.g., Nelson, Sweet, Berry, Bryant, & Granacher, 2007). In fact, malingering is likely best conceptualized as a continuous, rather than categorical, construct (e.g., Mossman, 2000). There may be no bright line which distinguishes malingering from nonmalingering. Performance on SVTs and PVTs does not fall into discrete groups or taxons (e.g., Armistead-Jehle et al., 2012; Lilienfeld et al., 2013; Walters et al., 2008). Base rates, sensitivity, and specificity in feigning assessment. Base rates of feigning vary significantly across assessment contexts (e.g., compensation-seeking versus non-compensation-seeking groups; Mittenberg, Patton, Canyock, & Condit, 2002). Basic conditional probability requires that SVT and PVT failure should not carry the same interpretation in all settings or with all populations (e.g., Bigler, 2012; also see Dean, Victor, Boone, Philpott, and Hess 2009, demonstrating poor SVT/PVT accuracy in dementia populations). It goes without saying that not all SVTs and PVTs are created equal : It is not uncommon for valid examinees (e.g., those with severe psychiatric disorders or moderate TBI) to fail SVTs/PVTs with lower specificity (e.g., Victor, Boone, Serpa, Buehler, & Ziegler, 2009). Conversely, feigning subjects can pass SVTs/PVTs with low sensitivity (e.g., Greve, Ord, Curtis, Bianchini, & Brennan, 2008). Varying clinical populations.. Some commonly encountered clinical populations, such as those with schizophrenia (Stevens et al., 2014), severe psychiatric illness (Hunt, Root, & Bascetta, 2013), PTSD (e.g., Merten, Thies, Schneider, & Stevens, 2009; Rubenzer, 2009), or moderate TBI (Greve et al., 2008), demonstrate significant failure rates on SVTs/PVTs even in the absence of potential compensation. In a recent study of Iraq War veterans, McCormick, Yoash-Gantz, McDonald, Campbell, and Tupler (2013) found false-positive rates on SVTs and PVTs of approximately 15% in subjects with no apparent incentive to exaggerate. Stevens et al. (2014) found a 25% failure rate on the WMT among schizophrenic patients who were not seeking compensation. Elevation on validity scales is common among PTSD patients, even among those not seeking compensation. Rubenzer (2009) conducted a meta-analysis of validity indices in PTSD and suggested the use of specific, more conservative cutting scores on the MMPI 84

8 impact of validity testing on expert opinion in order to minimize misclassification of PTSD examinees as feigning. Merten et al. (2009) utilized a combination of both SVTs and PVTs to identify potential symptom exaggeration among PTSD claimants. Test Selection In choosing which SVTs or PVTs to use in an assessment, experts must be able to articulate their a priori assumptions about each SVT and PVT they intend to use, including its base rate, sensitivity, and specificity (e.g., Mossman, 2000). For example, using an SVT or PVT with marginal specificity in a low base rate population can lead to erroneous decisions with high-stakes consequences. Currently, there are likely several dozen SVTs and PVTs that are in common use either for cognitive or psychological feigning. Many of these tests are embedded within typical neuropsychological measures, while others are freestanding and separate. Berthelson, Mulchan, Odland, Miller, and Mittenberg (2013) suggested that there are potential confounds among newer PVTs that could artificially elevate their false-positive rates. For ethical and validity concerns applied to high-stakes testing (e.g., American Educational Research Association [AERA], 1999), SVTs and PVTs should be selected according to validity and replication across several studies and different populations (e.g., comparing simulators, likely feigners, and clinical populations; Rogers, 2008). In addition to validity, selection of appropriate SVTs and PVTs should be guided by their sensitivity and specificity in the relevant population or domain. Because of the high-stakes nature of symptom validity testing, most authors have focused on specificity (i.e., minimizing false-positives) as their most important characteristic. In studies examining the latent sensitivity of several PVTs in the absence of a gold standard, Mossman, Wygant, and Gervais (2012) and Greve et al. (2008) found that several common PVTs are roughly equally accurate (i.e., have equivalent areas under the curve, or AUCs); however, they have significant trade-offs in sensitivity versus specificity. Both studies found the WMT significantly more sensitive than the TOMM, while the latter had substantially fewer false-positives (although see Greiffenstein, Greve, Bianchini, & Baker, 2008, for an alternate comparison). Whichever SVT or PVT the examiner chooses, it is incumbent upon the expert to be completely familiar with the test s characteristics for the population at hand. Because of the domain-specific nature of SVTs and PVTs, the types of measures administered should be tailored to the examinee s complaints and the type of objective tests employed. When forensic examinations are evaluating alleged impairments in both the cognitive and psychological domains, both SVTs and PVTs should be administered for adequate evaluation of feigning and effort (Nelson et al., 2007). Choosing the number of SVTs and PVTs to administer. It is commonly accepted practice to administer multiple PVTs throughout the course of assessment (e.g., Bush et al., 2005). Part of the rationale for this recommendation includes an ongoing assessment of potential changes in motivation. A second rationale is that the use of multiple tests allows examiners to chain likelihood ratios and be more confident in their conclusions of exaggeration or accurate responding (e.g., Larrabee, 2008). This may be true, but only under specific conditions of statistical independence among the tests, which may not hold true for most PVTs. 85

9 lockhart Potential problems with the administration of multiple PVTs are discussed by Berthelson et al. (2013), who conducted a meta-analysis of multiple PVTs in populations without any discernible incentive to malinger. They found a correlation of approximately 0.30 among tests, rather than statistical independence. Davis and Millis (2014) also found an average correlation above 0.30 among SVTs/PVTs. Berthelson et al. suggested that the typical false-positive rate of many SVTs is roughly 10% to 15%, rather than the 5% to 10% rate that is currently considered desirable. Berthelson et al. used this empirical data to develop a Monte Carlo simulation of how multiple PVTs will perform, especially with regard to false-positive outcomes. This work was recently replicated by Odland, Lammy, Martin, Grote, and Mittenberg (2015). They found that false-positive rates increase substantially as more tests are given, and must be interpreted within the context of overall PVT performance. For example, Berthelson et al. found a 20% chance that a nonmalingering subject will fail two out of five PVTs by chance. They suggested requiring failure on at least four out of five tests in order to keep the false-positive rate below 5%. In a recent study, Davis and Millis (2014) suggested that Berthelson et al. may have overestimated the false-positive rate. Even so, they found a 15% false-positive rate when using two failures out of six or more tests as the criterion for feigning. A recent study by Bashem et al. (2014) found an average specificity of 0.85 when two validity tests were given to non-compensation-seeking TBI patients from simulators, as well as significant intercorrelations among the validity tests. In practical terms, forensic experts should make an a priori determination about how they will interpret single or multiple SVT or PVT failure when multiple tests are administered. Odland et al. (2015) provide useful tables for interpreting multiple PVT failure in the context of total number of PVTs administered. Expressing Varying Levels of Certainty In a well-known article by Bazelon (1974), a jurist suggests that the trier-of-fact is best assisted when experts make clear the limitations of their opinion: [Psychologists and] psychiatrists should frankly explain on the witness stand that their opinions are thus... qualified by lack of time and resources. It was no service to the administration of justice for them to create the false impression that they had learned substantially all that could be known about someone on the basis of study they knew was inadequate (Bazelon, 1974, p. 20). Most of the time, experts must expect to operate with some level of uncertainty in their assessments of an examinee s accurate effort and symptom endorsement. It is rare that examiners will have incontrovertible evidence that an examinee is malingering for the purpose of specific gain. Iverson (2006) and Young (2014) have developed guidelines and models that allow for experts to express differing levels of certainty. The Slick et al. (1999) and Bianchini et al. (2005) models also include possible and probable categories, consistent with limitations on experts levels of certainty. As Iverson noted, examiners are ethically bound to describe their assessment results, including each element that suggests exaggeration versus honest responding. Young s (2014) classification system encourages experts to explicitly state the findings that tend to support or undermine the potential probability of malingering/feigning and allows for gray areas of malingering classification. Although both the APRD and MND address this issue in a limited fashion in their discussion of below-chance results, results that are beyond the cut-off but also 86

10 impact of validity testing on expert opinion represent better than chance responding deserve greater attention. To the extent that reliable data exist for differing levels of cut scores, evaluators can use such data to articulate their increased confidence that examinees are exaggerating their deficits. Avoiding Noncredible Explanations of SVT and PVT Failure Experts must describe evidence both for and against their hypotheses (Iverson, 2006). It is often suggested that exaggerated objective and self-report testing results and failure of SVT or PVT tests are caused by a third factor, such as depression or fatigue (e.g., Constant et al., 2011). However, research supporting this assertion is often weak or inconsistent (McClintock, Husain, Greer, & Cullum, 2010; Merten & Merckelbach, 2013; but, see Rubenzer 2009 for a discussion of SVT scores in PTSD). After excluding subjects who failed PVTs, Demakis et al. (2008) found no relation between PTSD scores and cognitive impairment. Rohling (2002) also found nonsignificant correlations between depression and cognitive functioning after excluding subjects failing PVTs. For a full discussion of noncredible explanations of seemingly feigned performance, the reader should consult several comprehensive sources on this topic, including Rogers (2008), Carone and Bush (2012) and Young (2014). Section Summary Several practical challenges exist in conducting validity testing with SVTs/PVTs. While there is significant potential in the slightly differing models of malingering detection and multiple tests available, there are also limitations. Chief among these limitations is the apparent likelihood of varying base rates, sensitivities, and specificities in different populations, and the possibility of increasing false-positives when using multiple SVTs/ PVTs. Some tentative suggestions include expressing conclusions probabilistically, rather than in absolute terms; avoiding explanations that are not empirically supported; and integrating collateral information. Determining, a priori, what the general range of PPV will be given the SVTs and/or PVTs administered, and how one will interpret SVT or PVT failure, should go a long way toward mitigating potential bias in after-the-fact (post hoc) judgments. Ethical and Legal Issues How should ethical forensic psychologists develop their expert opinions when examinees fail SVTs and PVTs, especially when an area of psychological functioning is a focus of legal concern? Answers may be found in the pertinent ethical guidelines, as well as the applicable legal decisions regarding admissibility of scientific testimony. Evaluators need to be familiar with all APA ethical guidelines (APA, 2010, 2013). Chief among these, the forensic psychologist has a duty to avoid doing harm. This is consistent with the APA ethical guidelines (APA, 2010) of Beneficence and Nonmaleficence : Psychologists strive to benefit those with whom they work and take care to do no harm (p. 3). In high-stakes testing using SVTs and PVTs, the most obvious harm would occur by calling an examinee a malingerer without adequate evidence. Unless examiners are confident of the base rate, sensitivity, specificity, and 87

11 lockhart positive predictive value (PPV) of the assessment, they should be cautious in using labels that carry such serious consequences. Integrating Data and Testing Rival Hypotheses An essential distinction between forensic and clinical evaluations is that rather than simply accepting the individual s self-report, forensic psychologists strive to corroborate their findings and conclusions. As discussed previously, PVT failure is associated with distortions on self-report of both current symptoms and premorbid functioning (the good old days effect). This potential bias in self-report can be tested by examining collateral information (e.g., medical records, financials, etc.). Forensic experts seek to gather adequate data to allow them to test rival hypotheses, rather than obtaining data that primarily supports their assumptions. This direction to seek corroborating data is found in the APA s Specialty guidelines for forensic psychology (SGFPA) and ethical guidelines (EPPCC). The SGFPA (section 9.02) states: Forensic practitioners ordinarily avoid relying solely on one source of data, and corroborate important data whenever feasible. When relying upon data that have not been corroborated, forensic practitioners seek to make known the uncorroborated status of the data, any associated strengths and limitations, and the reasons for relying upon the data. (APA, 2013, p. 15) The EPPCC (Section 9.01) similarly states that psychologists should have adequate data to substantiate their opinions: Psychologists base the opinions contained in their recommendations, reports, and diagnostic or evaluative statements, including forensic testimony, on information and techniques sufficient to substantiate [emphasis added] their findings. (APA, 2010, p. 12) The SGFPA guidelines encourage forensic evaluators to gather data that will not only substantiate their findings, but will also differentially test plausible rival hypotheses (SGFPA, 9.01). Similarly, Kane (2006) underscored the importance of seeking both corroborating and disconfirming information: It is essential that multiple methods and sources be utilized in the assessment, to try to ensure that all important factors are considered and that the data from each factor can be compared with those from the other factors, looking for convergent validity. (p. 27) At a fundamental level, SVT and PVT failure forces forensic evaluators to address two rival hypotheses: The first hypothesis is that the examinee validly performed the assessment tasks, and that the testing results are an accurate reflection of his or her true ability. The alternate hypothesis is that the evaluation results are not an accurate measure of the examinee s ability or symptoms, and that much of his or her history and testing results cannot be taken at face value. Moreover, the implication is that a better explanation of the data may involve negative response bias, distortion, feigning, or even malingering. If the alternate hypothesis is accepted, then the evaluation results and self-report will be deemed unreliable and should not be used to guide the expert s reasoning or 88

12 impact of validity testing on expert opinion conclusions. This loss of evaluation data may significantly limit the type of corroborating information available. The expert should examine the entire evaluation and context, and determine where the examinee best fits on the continuum of negative response bias given the available data. Legal/Evidentiary Guidelines From an evidentiary perspective, the distortions associated with SVT and/or PVT failure make the facts or data from the evaluation unreliable, in the sense that experts cannot confidently rely upon them as a basis for their opinions (Federal Rules of Evidence [FRE] 702; EPPCC; SGFPA). Similarly, facts that are unreliable cannot be combined with other evidence to form a sufficient foundation for an opinion. FRE 702 lays out the admissibility requirements that expert witnesses must meet, including the facts and data they use to form their opinions: (b) the testimony is based on sufficient facts or data; (c) the testimony is the product of reliable principles and methods; and (d) the expert has reliably applied the principles and methods to the facts of the case [emphasis added]. FRE 702(b) requires that the facts or data used by experts must be sufficient for the opinion they form. General Electric Co. v. Joiner (1997), one of the cases in the Daubert Trilogy, highlights the requirement for sufficiency of evidence in order for it to be admissible: A court may conclude that there is simply too great an analytical gap between the data and the opinion proffered... because it was within the District Court s discretion to conclude that the studies upon which the experts relied were not sufficient [emphasis added], whether individually or in combination, to support their conclusions. (p. 9) Section Summary Forensic psychologists are guided by the ethical principles of their profession. Because they function within the legal system, their assessments and conclusions must also meet the requirements for legal admissibility. A key tenet of their ethical and legal guidelines is that the information obtained through assessment must be both reliable and sufficient to support the conclusions reached in the evaluation. Although SVT and PVT failure makes it far more difficult to determine the actual source of causality in psychological injury, the total evidence may point to conclusions with an acceptable degree of certainty (i.e., more likely than not). Several authors (e.g., Iverson, 2006; Young, 2014) have suggested ways that forensic experts can communicate varying levels of certainty and causality about their findings and, therefore, be more useful to the trier-of-fact. 89

13 lockhart Concluding Remarks Forensic psychologists typically rely upon information from the clinical interview, objective testing, and third parties to generate and test rival hypotheses and form their conclusions. However, it is essential that evaluators have access to reliable data sufficiently adequate to this task. When an examinee fails pertinent SVTs and/or PVTs, the expert may no longer rely upon large portions of information that might have supported the examinee s claims of impairment. Examinees self-reports of current symptoms, their preinjury functioning, and their objective test results become unreliable and cannot be used to corroborate their self-reports or other data. This loss of reliable psychological evidence makes it far more difficult for the expert to confirm any claims. Moreover, significant SVT or PVT failure points to conclusions involving distortion, feigning, or even malingering, depending on the nature and complete data of the assessment. Because third-party data is so important after SVT and PVT failure, understanding how experts can best use this data would be helpful (e.g., Austin, 2002; Rogers, 2008, pp ). Similarly, clarification of the reliability and trade-offs of varying cut-scores for common SVTs and PVTs in different populations will help evaluators think more probabilistically when placing weight on validity test failure. Finally, the best ways to utilize multiple SVT and PVT failure data seem amenable to empirical and statistical investigation and deserve serious attention, since these validation tests represent a common testing approach. Following SVT or PVT failure, experts may be left with little reliable information from the sources outlined by Kane (2006) for example, third-party information. Forensic evaluators may be able to provide important nomothetic information (e.g., Heilbrun, Grisso, & Goldstein, 2008) related to general causality (e.g., in terms of base rates, empirical links in the literature between a specific injury and symptoms). However, they could be severely limited in their ability to draw specific idiographic conclusions regarding particular individuals (at least in terms of the nature of their disorders and disabilities). Ethically, experts need to communicate these limitations regarding evaluees claims and the increased probability of factors related to negative response bias. Along with expressing varying levels of certainty, experts should be able to clearly articulate each piece of psychological or other evidence (including passing or failing SVTs or PVTs) that tends to make examinees results more (or less) reliable. The job of forensic experts is not to advocate for a particular conclusion, but to assist triers-of-fact in making a decision. When data are limited or inconsistent, experts may do this through expressing varying levels of certainty. In the end, it may be that forensic evaluators cannot confidently say much more than that the examinee s results are unlikely to be an accurate reflection of his or her true abilities due to SVT or PVT failure, detailing the foundation of the opinion. Though this approach is perhaps less satisfying, experts must be careful not to go beyond the data into speculation. As Lilienfeld et al. (2013, p. 7) noted in a recent article on validity testing, The interpretation of neuropsychological assessment data hinges critically on valid data. The same is doubly true for forensic assessments, which carry high-stakes consequences and must be able to withstand judicial scrutiny. SVT and PVT failure has a broad impact on the type of data most relied upon by forensic psychologists, including objective testing, a clinical interview, and self-report. Significant failures on validity 90

14 impact of validity testing on expert opinion testing, although not proof of malingering, limit the data that experts can rely upon for the purpose of substantiating hypotheses and suggest lack of credibility in the examinee s approach to the assessment. This, in turn, narrows the conclusions that forensic psychologists may draw from an evaluation, and the information they may in turn provide to the trier of fact. Examiners, rather than focusing on diagnoses related to a putative psychological injury due to an event at claim, will need to focus on alternative explanations for noncredible presentation and performance, including symptom exaggeration, feigning, or even malingering. References American Educational Research Association. (1999). Standards for educational and psychological testing. Washington DC: Author. American Psychological Association. (2010) Amendments to the 2002 Ethical principles of psychologists and code of conduct. American Psychologist, 65 (5), doi: /a American Psychological Association. (2013). Specialty guidelines for forensic psychology. American Psychologist, 68 (1), Retrieved from doi: /a Armistead-Jehle, P., Gervais, R. O., & Green, P. (2011). Memory complaints inventory results as a function of symptom validity test performance. Archives of Clinical Neuropsychology, 27 (1), doi: /arclin/acr081 Armistead-Jehle, P., Gervais, R. O., & Green, P. (2012). Memory complaints inventory and symptom validity test performance in a clinical sample. Archives of Clinical Neuropsychology, 27 (7), doi: /arclin/acs071 Austin, W. G. (2002). Guidelines for utilizing collateral sources of information in child custody evaluations. Family Court Review, 40 (2), doi: /j tb00828.x Barsky, A. J. (2002). Forgetting, fabricating, and telescoping. Archives of Internal Medicine, 162 (9), doi: /archinte Bashem, J. R., Rapport, L. J., Miller, J. B., Hanks, R. A., Axelrod, B. N., & Millis, S. R. (2014). Comparisons of five performance validity indices in bona fide and simulated traumatic brain injury. The Clinical Neuropsychologist, 28 (5), doi: / Bazelon, D. L. (1974). Psychiatrists and the adversary process. Scientific American, 230, Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of consulting and clinical psychology, 56 (6), Bender, S. D., & Matusewicz, M. (2013). PCS, iatrogenic symptoms, and malingering following concussion. Psychological Injury and Law, 6 (2), doi: / s Berthelson, L., Mulchan, S. S., Odland, A. P., Miller, L. J., & Mittenberg, W. (2013). False positive diagnosis of malingering due to the use of multiple effort tests. Brain Injury, 27 (7 8), doi: /

15 lockhart Bianchini, K. J., Greve, K. W., & Glynn, G. (2005). On the diagnosis of malingered pain-related disability: Lessons from cognitive malingering research. The Spine Journal, 5 (4), doi: /j.spinee Bigler, E. D. (2012). Symptom validity testing, effort, and neuropsychological assessment. Journal of the International Neuropsychological Society, 18 (04), doi: /s Bush, S., Ruff, R., Troster, A., Barth, J., Koffler, S., Pliskin, N.,... Silver, C. (2005). Symptom validity assessment: Practice issues and medical necessitynan policy & planning committee. Archives of Clinical Neuropsychology, 20 (4), doi: /j.acn Butcher, J. N., Graham, J. R., Ben-Porath, Y. S., Tellegen, A., & Dahlstrom, W. G. (2003). MMPI-2: Minnesota Multiphasic Personality Inventory-2. Minneapolis: University of Minnesota Press. Carone, D. A., & Bush, S. S. (2012). Mild traumatic brain injury: Symptom validity assessment and malingering. New York: Springer Publishing Company. Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1978). Perceptual aberration scale. Retrieved from 20Aberration%20Scale.pdf Constant, E., Adam, S., Gillain, B., Lambert, M., Masquelier, E., & Seron, X. (2011). Cognitive deficits in patients with chronic fatigue syndrome compared to those with major depressive disorder and healthy controls. Clinical Neurology and Neurosurgery, 113 (4), doi: /j.clineuro Davis, J. J., & Millis, S. R. (2014). Examination of performance validity test failure in relation to number of tests administered. The Clinical Neuropsychologist, 28 (2), doi: / Dean, A. C., Victor, T. L., Boone, K. B., Philpott, L. M., & Hess, R. A. (2009). Dementia and effort test performance. The Clinical Neuropsychologist, 23 (1), doi: / DeClue, G. (2002). Feigning deos not equal malingering: A case study. Behavioral Sciences & the Law, 20 (6), doi: /bsl.490 Demakis, G. J., Gervais, R. O., & Rohling, M. L. (2008). The effect of failure on cognitive and psychological symptom validity tests in litigants with symptoms of post-traumatic stress disorder. The Clinical Neuropsychologist, 22 (5), doi: / Ferguson, R. J., Mittenberg, W., Barone, D. F., & Schneider, B. (1999). Postconcussion syndrome following sports-related head injury: Expectation as etiology. Neuropsychology, 13 (4), doi: / Forbey, J. D., Lee, T. T. C., Ben-Porath, Y. S., Arbisi, P. A., & Gartland, D. (2013). Associations between MMPI-2-RF validity scale scores and extra-test measures of personality and psychopathology. Assessment, 20 (4), doi: / Frederick, R. ((1997)). Validity indicator profile test manual. Minneapolis, MN: National Computer Systems. Grant, D. A., & Berg, E. A. (1993). Wisconsin card sorting test (WCST). Odessa, FL: Psychological Assessment Resources. Green, P., & Allen, L. (2004). Memory complaints inventory. Edmonton, Canada: Green s Publishing. 92

16 impact of validity testing on expert opinion Green, P., Allen, L., & Astner, K. (1997). The word memory test: A manual for the oral and computerized forms. Durham, NC. Green, P., Rohling, M. L., Lees-Haley, P. R., & III, L. M. A. (2001). Effort has a greater effect on test scores than severe brain injury in compensation claimants. Brain Injury, 15 (12), doi: / Greiffenstein, M., Greve, K. W., Bianchini, K. J., & Baker, W. (2008). Test of memory malingering and word memory test: A new comparison of failure concordance rates. Archives of Clinical Neuropsychology, 23 (7-8), doi: /j.acn Greve, K. W., Ord, J., Curtis, K. L., Bianchini, K. J., & Brennan, A. (2008). Detecting Malingering in traumatic brain injury and chronic pain: A comparison of three forced-choice symptom validity tests. The Clinical Neuropsychologist, 22 (5), doi: / Gunstad, J., & Suhr, J. A. (2001). expectation as etiology versus the good old days : Postconcussion syndrome symptom reporting in athletes, headache sufferers, and depressed individuals. Journal of the International Neuropsychological Society: JINS, 7 (3), doi: /s Gunstad, J., & Suhr, J. A. (2004). Cognitive factors in postconcussion syndrome symptom report. Archives of Clinical Neuropsychology, 19 (3), doi: /s (03) Heilbronner, R., Sweet, J., Morgan, J., Larrabee, G., Millis, S., & Participants, C. (2009). American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and Malingering. The Clinical Neuropsychologist, 23 (7), doi: / Heilbrun, K., Grisso, T., & Goldstein, A. M. (2008). Foundations of forensic mental health assessment. New York: Oxford University Press. Hunt, S., Root, J. C., & Bascetta, B. L. (2013). Effort testing in schizophrenia and schizoaffective disorder: Validity indicator profile and test of memory Malingering performance characteristics. Archives of Clinical Neuropsychology, 29 (2), doi: /arclin/act069 Iverson, G. L. (2006). Ethical issues associated with the assessment of exaggeration, poor effort, and Malingering. Applied Neuropsychology, 13 (2), doi: / s an Iverson, G. L., & Lange, R. T. (2003). Examination of postconcussion-like symptoms in a healthy sample. Applied Neuropsychology, 10 (3), doi: / s an Iverson, G. L., Lange, R. T., Brooks, B. L., & Lynn Ashton Rennison, V. (2010). good old days bias following mild traumatic brain injury. The Clinical Neuropsychologist, 24 (1), doi: / Kane, A. W. (2006). Psychological knowledge in court: PTSD, pain, and TBI. In G. Young, A. Kane, & K. Nicholson (Eds.), (pp ). New York: Springer. Larrabee, G. J. (2008). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. The Clinical Neuropsychologist, 22 (4), doi: / Lilienfeld, S. O., Thames, A. D., & Watts, A. L. (2013). Symptom validity testing: Unresolved questions, future directions. Journal of Experimental Psychopathology, 4 (1), doi: /jep