Advances in Measuring Non-Cognitive Skills Tim Kautz Mathematica Policy Research RISE Conference June 17, 2015 This draft, June 13, 2016 Kautz Inequality 1 / 32
Measuring Non-Cognitive Skills We have come to focus on achievement test scores to assess students, teachers, and schools (and even countries) Achievement tests miss important non-cognitive skills that can be shaped through interventions and required for success in school and beyond (Kautz, Heckman, Diris, ter Weel, and Borghans, 2014) Key policy-makers and organizations are planning to measure non-cognitive skills at a large-scale but there are challenges Recent advances in measurement provide some solutions to these challenges relevant for international use Kautz Inequality 2 / 32
Outline 1 Why measure non-cognitive skills? 2 Traditional approaches to measuring non-cognitive skills 3 Challenges and advances in measuring non-cognitive skills Kautz Inequality 3 / 32
Source(s) James J. Heckman, John E. Humphries, and Tim Kautz (2014). The Myth of Achievement Tests: The GED and the Role of Character in American Life. Chicago, IL: University of Chicago Press. Kautz Inequality 4 / 32
Source(s) Fosteringzandz MeasuringzSkills:z ImprovingzCognitivez andznon-cognitivezskillsztoz PromotezLifetimezSuccess TimzKautz,zJameszJ.zHeckman,zRonzDiris,z BaszterzWeel,zLexzBorghans Tim Kautz, James J. Heckman, Ron Diris, Bas ter Weel, Lex Borghans (2014). Fostering and Measuring Skills Improving Cognitive and Non-cognitive Skills to Promote Lifetime Success. OECD Education Report. Kautz Inequality 5 / 32
1. Why measure non-cognitive skills? Kautz Inequality 6 / 32
There are three main reasons to measure non-cognitive skills 1 IQ and achievement tests miss important skills 2 Non-cognitive skills predict important long-term outcomes 3 Non-cognitive skills are malleable and can be improved through interventions Kautz Inequality 7 / 32
(1) IQ and Achievement tests miss non-cognitive skills We have come to place great emphasis on cognitive tests (Heckman and Kautz, 2014) But these tests are not all that predictive of later life outcomes Kautz Inequality 8 / 32
Figure 1: Validities of Cognitive Measures in Age-35 Labor Market Outcomes (Adjusted R-Squared) (a) Males Adjusted R Squared 0.05.1.15.2 Earnings Hourly Wage Hours Worked IQ AFQT GPA IQ AFQT GPA IQ AFQT GPA Source: Heckman and Kautz (2012) using the National Longitudinal Survey of Youth (NLSY79) Kautz Inequality 9 / 32
(2) Non-cognitive skills predict important life outcomes A common measurement system is the Big Five (Openness to Experience, Conscientiousness, Extraversion, Agreeableness, Neuroticism) Conscientiousness the tendency to be organized and complete tasks is the most predictive across a wide variety of outcomes Kautz Inequality 10 / 32
Figure 2: Association of the Big Five and Intelligence with Years of Schooling in GSOEP (a) Males Kautz Inequality 11 / 32
Hanushek_2011 978-0-444-53444-6 Figure 3: Correlations of Mortality with Non-Cognitive Skills, IQ, and Socioeconomic Status (SES) Personality Psychology and Economics 113 0.30 0.25 0.20 Correlation 0.15 0.10 0.05 0 SES IQ C E/PE N A 5 Figure 1.18 Correlations of Mortality with Personality, IQ, and Socioeconomic Status (SES). Source: Notes: Roberts, The figure Kuncel, represents Shiner, results from Caspi, a meta-analysis and Goldberg of 34(2007). studies. Average effects (in the correlation metric) of low socioeconomic status (SES), low IQ, low Conscientiousness (C), low Extraversion/Positive Emotion (E/PE), Neuroticism (N), and low Agreeableness (A) on mortality. Error bars represent standard Kautz Inequality 12 / 32
(3) Non-cognitive skills are malleable and can be improved through interventions IQ becomes relatively rank stable by age 10 while non-cognitive skills are more variable (Almlund, Duckworth, Heckman, and Kautz, 2011) Neuroscience shows that this malleability is associated with the slow development of the prefrontal cortex (Walsh, 2005) Consistent with review of the intervention literature (Kautz, Heckman, Diris, ter Weel, and Borghans, 2014) Only interventions that started before age 3 had a long-term effect on IQ Many interventions starting after age 3 have effectively improved outcomes by improving non-cognitive skills Adolescent interventions that teach personality skills in the workplace (or specific context) are promising Kautz Inequality 13 / 32
2. Traditional approaches to measuring non-cognitive skills Kautz Inequality 14 / 32
Non-cognitive skills are typically measured using self reports Self-reports are a primary method of measuring non-cognitive skills The Big Five is a relatively well-accepted taxonomy Kautz Inequality 15 / 32
Table 1: The Big Five Traits OCEAN Trait Definition of Trait I. Openness to Experience The tendency to be open to new aesthetic, cultural, or intellectual experiences. II. Conscientiousness The tendency to be organized, responsible, and hardworking. III. Extraversion An orientation of one s interests and energies toward the outer world of people and things rather than the inner world of subjective experience; characterized by positive affect and sociability. IV. Agreeableness The tendency to act in a cooperative, unselfish manner. V. Neuroticism Neuroticism is a chronic level of emotional instability and proneness to psychological distress. Emotional stability is predictability and consistency in emotional reactions, with absence of rapid mood changes. Kautz Inequality 16 / 32
Example question used to measure Conscientiousness I see myself as someone who tends to be lazy Rating scale: 1 strongly agree, 5 strongly disagree Kautz Inequality 17 / 32
3. Challenges and advances in measuring non-cognitive skills Kautz Inequality 18 / 32
Basic measurement challenges All psychological measurements are based on performance on a task (Heckman and Kautz, 2012) An interpretive problem lies at the heart of any psychological measurement system for any particular trait It is necessary to standardize for incentives and the effects of other traits in performing a task Kautz Inequality 19 / 32
Figure 4: Determinants of Task Performance Incentives Effort Non-Cognitive Character Skills Task Performance Cognitive Skills Test Scores Self-Reports Other Behaviors Kautz Inequality 20 / 32
Figure 5: Decomposing Variance Explained for Achievement Tests and Grades into IQ and Character: Stella Maris Secondary School, Maastricht, Holland Source: Borghans, Golsteyn, Heckman, and Humphries (2011). Note: Grit is a measure of persistence on tasks (Duckworth, Peterson, Matthews, and Kelly, 2007). Kautz Inequality 21 / 32
Can boost IQ by 15 points by giving candies for correct answers the Black/White gap in IQ in U.S Segal (2012) shows that introducing performance-based cash incentives in a low-stakes administration of a measure of IQ increases performance substantially among roughly one-third of participants. Kautz Inequality 22 / 32
Reference bias is one challenge with self-reports Respondents rate themselves relative to their peers rather than the population at large Reference bias can be especially problematic if comparing across different contexts (e.g. between countries and schools) A form of a situation (peer group) affecting measurement Kautz Inequality 23 / 32
Figure 6: National Rank in Big Five Conscientiousness and Average Annual Hours Worked Rank in Big Five Conscien7ousness (High to Low) Finland 1 United States 2 Chile 3 France 4 Slovenia 5 Canada 6 Turkey 7 Italy 8 Portugal 9 Greece 10 United Kingdom 11 Spain 12 Austria 13 Germany 14 Poland 15 Australia 16 Estonia 17 Mexico 18 Switzerland 19 New Zealand 20 Netherlands 21 Belgium 22 Czech Republic 23 Slovakia 24 South Korea 25 Japan 26 1300 1500 1700 1900 2100 2300 2500 Annual Hours Worked Source: The Conscientiousness ranks come from Schmitt, Allik, McCrae, and Benet-Martínez (2007). These measures were taken in 2001 (Schmitt, 2002). The hours worked estimates come from Organisation of Economic Cooperation and Development (2001). Notes: Several countries are omitted due to lack of data. Kautz Inequality 24 / 32
Anchoring vignettes address reference bias Anchoring vignettes are an additional question that helps to standardize the situation Describe a situation and ask the respondent to rate performance Kautz Inequality 25 / 32
Example anchoring vignette from Primi, Zanon, Santos, De Fruyt, and John (2016) used in Brazil Aline leaves her belongings in a mess, hates cleaning the house, and usually doesn t complete her homework. How organized do you think Aline is? (1) Not at all (2) A little (3) Moderately (4) Very much (5) Completely Kautz Inequality 26 / 32
Anchoring vignettes can be effective Kyllonen and Bertling (2013) demonstrates that including anchoring vignettes changes a cross-country relationship between teacher support and achievement from -0.45 to 0.29 in PISA 2012 Kautz Inequality 27 / 32
Figure 7: Determinants of Task Performance Incentives Effort Non-Cognitive Character Skills Task Performance Cognitive Skills Test Scores Self-Reports Other Behaviors Kautz Inequality 28 / 32
Explore alternative measures available to schools Measures traditionally viewed as outcomes contain information on non-cognitive skills Real-world non-cognitive skill measures: grades, absences, credits earned, disciplinary infractions Highly predictive of later behavior Kautz Inequality 29 / 32
Table 2: Predictive Validity (R 2 ) from Ninth-Grade Measures on Various Outcomes Outcome Ninth-Grade Measure Explore Test GPA Credits Absences Discipline All ACT Score (Grade 11) 0.78 0.22 0.05 0.10 0.02 0.79 GPA (Grade 11) 0.21 0.49 0.28 0.20 0.05 0.52 Absences (Grade 11) 0.09 0.22 0.12 0.35 0.03 0.39 Arrested within 4 Years 0.06 0.14 0.12 0.10 0.10 0.20 Grad HS within 5 Years 0.11 0.35 0.36 0.23 0.06 0.41 Enroll College within 6 Years 0.15 0.20 0.16 0.12 0.03 0.25 Grad College within 10 Years 0.17 0.17 0.07 0.09 0.01 0.23 Kautz Inequality 30 / 32
Figure 8: Predictive Validity of Cognitive and Non-Cognitive Skill for High School Graduation Grad HS within 5 Years R Squared 0.1.2.3.4 Cognitive Non Cognitive Total Measure Kautz Inequality 31 / 32
Conclusions Non-cognitive skills predict outcomes and are malleable There are challenges with implementing traditional measures of non-cognitive skills at scale Recent advances suggest some promising methods to address the challenges Kautz Inequality 32 / 32
Almlund, M., A. Duckworth, J. J. Heckman, and T. Kautz (2011): Personality Psychology and Economics, in Handbook of the Economics of Education, ed. by E. A. Hanushek, S. Machin, and L. Wößmann, vol. 4, pp. 1 181. Elsevier, Amsterdam. Borghans, L., B. H. H. Golsteyn, J. Heckman, and J. E. Humphries (2011): Identification Problems in Personality Psychology, Personality and Individual Differences, 51(3: Special Issue on Personality and Economics), 315 320. Duckworth, A. L., C. Peterson, M. D. Matthews, and D. R. Kelly (2007): Grit: Perseverance and Passion for Long-Term Goals, Journal of Personality and Social Psychology, 92(6), 1087 1101. Heckman, J. J., and T. Kautz (2012): Hard Evidence on Soft Skills, Labour Economics, 19(4), 451 464, Adam Smith Lecture. (2014): Achievement Tests and the Role of Character in American Life, in The Myth of Achievement Tests: The GED and the Role of Character in American Life, ed. by J. J. Heckman, J. E. Humphries, and T. Kautz, pp. 3 56. University of Chicago Press, Chicago, IL. Kautz, T., J. J. Heckman, R. Diris, B. ter Weel, and L. Borghans (2014): Fostering and Measuring Skills: Improving Cognitive and Non-Cognitive Skills to Promote Lifetime Success, OECD, Forthcoming. Kyllonen, P. C., and J. P. Bertling (2013): Innovative questionnaire assessment methods to increase cross-country comparability, Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis, 277. Organisation of Economic Cooperation and Development (2001): OECD Employment and Labour Market Statistics, Data available from http://www.oecd-ilibrary.org/employment/data/ oecd-employment-and-labour-market-statistics_lfs-data-en. Kautz Inequality 32 / 32
Primi, R., C. Zanon, D. Santos, F. De Fruyt, and O. P. John (2016): Anchoring Vignettes, European Journal of Psychological Assessment. Roberts, B. W., N. R. Kuncel, R. L. Shiner, A. Caspi, and L. R. Goldberg (2007): The power of personality: The comparative validity of personality traits, socioeconomic status, and cognitive ability for predicting important life outcomes, Perspectives in Psychological Science, 2(4), 313 345. Schmitt, D. P. (2002): Are Sexual Promiscuity and Relationship Infidelity Linked to Different Personality Traits across Cultures? Findings from The International Sexuality Description Project, Online Readings in Psychology and Culture, 4(Unit 4), 1 22, Retrieved from http://scholarworks.gvsu.edu/orpc/vol4/iss4/4. Schmitt, D. P., J. Allik, R. R. McCrae, and V. Benet-Martínez (2007): The Geographic Distribution of Big Five Personality Traits: Patterns and Profiles of Human Self-Description Across 56 Nations, Journal of Cross-Cultural Psychology, 38(2), 173 212. Segal, C. (2012): Working When No One Is Watching: Motivation, Test Scores, and Economic Success, Management Science, 58(8), 1438 1457. Walsh, D. A. (2005): Why Do They Act That Way? A Survival Guide to the Adolescent Brain for You and Your Teen. Free Press, New York, 1 edn. Kautz Inequality 32 / 32