The Dunning-Kruger Effect in Professional Psychology

Transcription

1 The Dunning-Kruger Effect in Professional Psychology John Caccavale, Ph.D. The 1999 paper Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments by David Dunning an Justin Kruger was an instant success among talk show hosts and comedians. 1 In the first few years after the paper was published, studies started to appear in many professional journals. The problematic aspect of incompetence, of course, is that you have a group of people who are completely unaware of their incompetence and, when considering the cognitive dissonance factor, they dig in their heels to justify their position. Clearly, it doesn't take much to see the Dunning-Kruger effect at work in many everyday interactions. However, I would like to expand on their work to point out a somewhat different slant on the issue. It's one thing to study how stupid and ignorant people are and the irrational conclusions they formulate and act upon; it's another matter, however, when we see trained professionals and the highly educated offering opinions and policies that make them appear like stupid people trying to pass as smart ones. In a perverse restatement of Milton Erickson's view of the unconscious, "They're incompetent but they don't know they're incompetent." In an interview 2 after their paper was published, Dunning stated, "An ignorant mind is precisely not a spotless, empty vessel, but one that s filled with the clutter of irrelevant or misleading life experiences, theories, facts, intuitions, strategies, algorithms, heuristics, metaphors, and hunches that regrettably have the look and feel of useful and accurate knowledge." If we look closely enough and without the colored glasses of our own subjective feelings and long held beliefs, everyone, from time to time, can articulate some really stupid connections about the world we live in. The Dunning-Kruger effect should not be about isolated stupidity or the pseudo-philosophical rants by the Duck Dynasty, for example. It's should not be confined to the ranks of the less educated. Our concern should be about those who we believe are competent and are in a position to make or influence policies and important decisions that affect our lives and all of society. For example, what do we really know about the professionals that we rely upon who make judgements about our healthcare? What about policymakers whose policies have the potential to produce harm? In our present environment of the security state, what about our military leaders, do we know how much knowledge and what that knowledge is based upon when their recommendations are acted upon? As for many politicians, I really do not think we are in the dark about them. For the past many years we have heard and come to accept that, in politics, ignorance is bliss. But what about psychology, I haven't seen much, if anything, about the D-K effect being applied to our profession. To that end, I would like to offer a few areas where I believe the D-K effect reigns supreme in the policies and practices that we have all come to accept as professionals psychologists. Please note: The examples are not finite.

2 The Belief In Evidence-Based Therapy There is no question that striving to determine and implement treatments that work is a needed and important goal. But, seeking is not the issue. The salient issue is how does one determine what is and what is not evidenced based. This issue addresses research design, reproducibility of results, significance and, most importantly, clinical relevance. As it stands, it is my contention that for psychology determining what is an evidencebased therapy may be as elusive as the quest for the Holy Grail. This conclusion is not based upon a belief that the subject matter of psychological therapy is in the realm of the "soft" sciences. On the contrary, our subject matter is in the realm of really hard science. Physical and chemical research is easier to do because a molecule of carbon and a ray of light can be seen and studied similarly anywhere. They are variables that can be manipulated because of consistency and predictability. Human behavior, however, is not consistent or very predictable. Also, ethical considerations make manipulation of human beings unacceptable. So, whose science is really soft? The research designs that must be used to determine what is and what is not evidenced based is not available to psychologists; yet, there are those in our profession who seem intent on promoting something utilizing babble not based upon a competent understanding of the issues. Mostly, the pandering to EBT is nothing more than trying to get reimbursement for favored therapies. In medicine, for example, few studies looking at the effectiveness of medications have been able to be reproduced. Oncologist researchers who analyzed the many efficacious cancer therapies reported that scientists at the pharmaceutical company Amgen could not replicate the overall majority of published pre-clinical research studies. Only 6 out of 53 of the most important cancer studies could be replicated. A success rate of 11%. 3 Moreover, the problem of reproducibility is present in many other areas of medicine. So, if evidenced-based therapies are difficult to find in medicine, what is the potential for evidence based therapies in psychology? When having a patient sit in a corner going through a manual is the most exciting evidenced-based therapy and having our profession continue to spew out rhetoric about it, the Dunning-Kruger effect is a good way to address the denial of reimbursement from insurers. With respect to significance, for too long we have been saddled with the rule.05 and.01 rules for determining significance. Medicine has no problem getting treatments approved for therapies that show only 30% clinical significance. I contend that professional psychology and patients would gain if we turned our focus to the more relevant and truthful criterion embedded in clinical significance. Our unwise commitment to statistical processes and the "truth" that it imparts simply is not justified or competent in evaluating clinical therapies. There is a place for statistical significance and we should develop the guidelines that make sense and not simply bow down to numbers that most do not understand in terms of relevance to patient improvement. Lastly, what we do know about what works is related to whose doing the treatment. The therapist variable has been demonstrated to be the most explanatory variable in assessing patient improvement. This one variable is both clinically and statistically significant. I suggest that we need to seriously consider the Dunning-Kruger effect as another metric when looking at evidenced-based therapy.

3 Does Accreditation Assure Quality? Since 1940 when APA began promoting accreditation of psychology programs psychologists, employers and government agencies have accepted that accreditation implies or insures quality. The APA website contains numerous statements that they say support this claim. Yet, there is not one single study that demonstrates psychologists who are graduated from APA accredited programs provide more quality services than psychologists graduated from non-accredited programs. There is not one study that demonstrates graduates from APA programs are more competent, are less likely to be disciplined or have their licenses revoked or restricted because of incompetence. What can be said about APA accreditation is that graduates from APA accredited programs have more job opportunities because APA has been successful in convincing government agencies and other employers that there is some value in APA accreditation. Without further elaboration, statements about APA accreditation and its value are simply not in the realm of competent analyses. One reason for this lack of evidence is that APA has not thought it necessary to do the research to justify its claims because no research would show that APA accreditation does anything more than increase the tuition of psychology programs. Until professional psychology agrees upon a standardized program of training, as does law and medicine and many other professions, accreditation claims will fall under the Dunning-Kruger effect. Licensing and The Psychology Testing Industry Every psychologist seeking to practice must take and successfully pass the Examination for the Professional Practice in Psychology (EPPP). The EPPP was developed by the ASPPB to assist the states when evaluating applicants in granting licenses to psychologists. Clearly, the assumption is that there is a relationship between minimal competence and a passing grade on the test. On its website the ASPPB states that the EPPP has high reliability. Yet, while there are some studies that seek to provide reliability and content validation to the EPPP, they report inconsistency and not very encouraging results about either the test's reliability, validity or of its contents. 4-5 This is important because deep in the culture of professional psychology is this strong belief in testing as a way to assure minimal competency. As psychologists we are so into this belief that we not only spend tens of millions of dollars in paying to take the EPPP, but untold millions more paying companies preparing us to pass the exam. While I haven't done a study on a state-by-state pass rate for the EPPP or any historical analysis, the ASPPB website states that it is a myth that there is a low pass rate on the exam. I do, however, have the latest results for California. In their latest newsletter, the California BOP states that the pass rate for the fiscal year was 62%. 6 Doesn't seem like an overwhelmingly high rate. In fact, since the EPPP purports to test the minimal knowledge that a doctoral level psychologist should possess, one may conclude that the schools preparing these psychologists are doing a really poor job. We may also conclude that APA accredited programs and their standards have little relationship to quality education and training. Lastly, we might conclude that the EPPP is really an expensive but relatively worthless way to evaluate minimal competence. Perhaps all three conclusions are correct.

4 The Dunning-Kruger effect here is not about stupid people or only incompetence. It's about rigidity of institutions and organizations to change and admit failure. It's about repeating beliefs not supported by the facts. This is classic Dunning-Kruger. It's also about economics both at the educational level, training level, the organization level, and the relationship between bloated salaries earned by staff in professional organizations and the associated legions of others making money off unsupported validity and reliability of the EPPP. I might add that psychology is not the only profession that needs to evaluate its testing mythology. Recently, the deans of some of the most prestigious law schools are questioning the reliability and validity of having law graduates take and pass the state bar to practice law. They see the bar exam as not representing the knowledge attorneys need to practice or more predictive than successfully completing law school. To visualize just how absurd are the claims that the EPPP is related or even necessary for licensure, the following table may provide some guidance. Establishing Minimal Competence for Licensing Psychologists THIS or THOSE plus THIS years post graduate study. Scored a minimum of 70% on the EPPP. 2. Successfully passed all coursework with a 3.0 or better. 3. Successfully submitted a dissertation or similar research project. 4. Successfully defended the dissertation. 5. Successfully passed content examinations. 6. Successfully completed a minimum of 3000 hours of supervised practice. Concluding Statements In conclusion, David Dunning's statement should be very worrisome to psychologists when evaluating what comes out of those in psychology who decide what and how knowledge in psychology should be acquired, practiced, regulated and evaluated. We must really look at the rational behind every claim made by those who purport to represent our interests and the interests of our patients.

5 References 1. Kruger, J. & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, Vol 77(6), Dec 1999, David Dunning Begley, C & Ellis, LM. (2012). Drug development: Raise standards for preclinical cancer research. Nature 483, (29 March 2012) doi: /483531a. 4. Sharpless, Brian A.; Barber, Jacques P.(2009. The Examination for Professional Practice in Psychology (EPPP) in the era of evidence-based practice. Professional Psychology: Research and Practice, Vol 40(4), Aug 2009, Sharpless, Brian A.; Barber, Jacques P. (2013). Predictors of program performance on the Examination for Professional Practice in Psychology (EPPP). Professional Psychology: Research and Practice, Vol 44(4), Aug 2013, The California Board of Psychology. Issue #4. Winter 2015.