Opening the Black Box - How Do Personality Traits. and Preferences Affect Test Scores?

Transcription

1 Opening the Black Box - How Do Personality Traits and Preferences Affect Test Scores? Benedikt Vogt Maastricht University b.vogt@maastrichtuniversity.nl Very preliminary draft. Please do not cite or circle without the author s permission. Abstract We experimentally investigate behavior during a test. Our experimental design allows us to distinguish between the speed with which a test taker knows a correct answer over time and when she submits that answer. The idea is that both speed of thinking and the decision when to submit an answer determine the final test result. We find that personality traits such as openness and neuroticism and an individual s risk preference influence the speed of thinking during a test. However, only an individual s discount rate determines the submission of an answer.

2 1. Introduction Standardized tests are the most frequently used tool to assess students, job applicants or to compare the educational performance of countries. It is well known that test results are shaped by cognitive and non-cognitive skills. Many studies find that personality traits such as conscientiousness and neuroticism and economic preferences such as time and risk preferences are associated with the test result. However, none of the current studies investigates the mechanism behind these relationship. The aim of this paper is to investigate how personality traits and preferences influence a test result. The idea is that there are two determinants of a test result. The first determinant is the probability of knowing the write answer while answering a question. The speed of coming up with a correct answer can also be described as the actual smartness of the test taker. The second determinant of a test result is the answering behavior of a test taker. Despite the mere speed of thinking the decision when to answer a question is a crucial determinant of the test result. We argue that personality traits and economic preferences can influence both determinants. First, they can influence the probability of knowing the correct answer at any point in time. Neurotic people, for instance, could in general be less likely to answer a test question correctly at any point in time. Second, personality traits could influence the behavior during a test. Assume that patient and less impatient people are in general all the time equally likely to know the answer when they answer a test question. This would mean that they in general equally smart. However, patient people could be more eager to invest more time in answering a question and hence are also more likely to have the correct answer on this question. This means that different personality types exhibit different behavior while answering a test. In a laboratory experiment each participant has to answer a set of questions of an intelligence test. While answering a test question subjects are faced with two independent payment 1

3 schemes. The first payment scheme serves as a thought-tracker. It provides an incentive to immediately reveal the answer which the subject thinks is the correct one at any point in time. This allows us to plot the probability of knowing the correct answer over time while answering a question. The second incentive scheme provides subjects with an incentive to come to a final decision on a question. This is the decision one usually observes in tests. Thus, these two payment scheme allow us to disentangle the smartness from the behavior during a test. We also measure personality with the Big Five inventory as well as Grit and Ambition. Moreover, we have measures of an individual s risk attitude and time preference. Investigating the mechanisms of how personality traits and preferences influence a test result is important for two reasons. First, if one knows why certain people score worse on tests, one can train them better. Second, solving a test can be seen as a proxy for problem solving in general. If one knows the mechanisms why different personality types perform differently in these tasks one can either train them also on these tasks or select them into different tasks. Our main findings can be summarized as follows. We find that openness and neuroticism influence the speed of thinking of a test taker. Moreover, risk lovers have a higher probability of knowing the answer to a test question over time. However, only an individual s discount rate determines the point in time when a subject submits the answer to a question. Our paper adds to the literature both in economics and psychology. Numerous papers identify a relationship between personality traits and test results. Borghans et al 2008 and Duckworth et al 2011 show that conscientiousness and economic preferences are correlated with test outcomes on intelligence and achievement tests. Dohmen et al 2010 find that risk aversion and impatience are negatively related to measures on a cognitive test. However, it is still not clear why certain personality types perform better on tests than others. 2

4 The remainder of the paper is structured as follows. Section 2 briefly summarizes the most important features of the experimental design. Section 3 presents the results and Section 4 concludes. 3

5 2. Experimental Design The first part of the experiment started with a detailed questionnaire on personality traits and economic preferences. We measured the Big Five (Goldberg 1990) using the IPIP. 1 We further measured Grit and Ambition using the questionnaire proposed by Duckworth et al Economic preferences such as risk attitude and time preferences were measured with experimentally validated items by Falk et al In the next part subjects received detailed instructions about the questions in the part of the cognitive test. Each subject had to answer 45 Raven matrices with different degrees of difficulty. Subjects had to face each test question for 60 seconds. 2 The order of the questions was randomized between subjects. Before they could start with the experiment, subjects had to go through a trial phase which made them familiar with the way we set up the test and the two payment schemes when answering a test question. During answering a question subjects were confronted with two independent payment schemes. The first payment scheme which we call the blue payment scheme, serves as a thoughttracker. It provides an incentive to immediately reveal the answer to a question which the subject thinks is the correct one at any point in time. More specifically we paid 0.5 cents for every second a subject selected the correct answer during the 60 seconds of answering a test question. Subjects could change their answer as often as they liked. The second incentive scheme which we called the red payment scheme was running independent of the first one. We provided subjects with an incentive to submit an answer by pressing a submit button. In the red payment scheme we varied the incentives to submit an answer in three treatments while the blue payment scheme remained constant over the whole experiment. In our baseline treatment the amount a subject could receive for submitting a correct answer decreased from 25 cents to 1 This test is available at 2 A screenshot of the decision making screen can be found in the appendix. 4

6 5 cents during the 60 seconds. We call this the LL treatment which stands for Low incentives and Low time pressure. In the second treatment which we call the HH treatment (High incentives and High time pressure) the reward for submitting a correct answer decreased from 55 cents to 5 cents during the 60 seconds. In the third treatment which we call the HL treatment (High incentives Low time pressure) the reward for submitting a correct answer decreased from 55 cents to 35 cents. Each treatment contained 15 questions in a randomized order. We randomized the treatment order across subjects. More information on the data and the procedure can be found in Borghans et al Results In this section we present the results of the experiment. The experiment was computerized with ztree (Fischbacher, 2007) and a total of 128 subjects participated. The basic idea is that differences in tests can arise from two reasons. The first one is that, differences in test outcome can be due to the fact that different personalities invest more or less effort when they think about a question. This has an impact on the probability of knowing the correct answer of a test question over time. The second reason is that, different personalities might differ not only how intense they think about a question but also how long they think about a certain test question Correlation between Preferences and Personality Traits Table 1 shows the relationship between the measures of the personality traits and the economic preference parameters. Within the Big Five taxonomy only Openness to Experience seems to be positively correlated with Extraversion and Conscientiousness. Ambition is significantly positively correlated with Openness, Conscientiousness and Extraversion. Grit is highly significantly positively correlated with Openness, Conscientiousness and Ambition. Grit and Neuroticism are highly significantly negatively correlated. Interestingly Neuroticism is significantly negatively correlated with the willingness to take risks. There seems to be a 5

7 positive relationship between an individual s risk attitude and Openness and Ambition. Most strikingly an individual s discount rate does not significantly correlate with any of the other personality and preference measures. 6

8 Conscientiousness 0.186* Table 1: Correlation Structure of Personality Traits, Preferences and Gender Openness Conscientiousness. Extraversion Agreeableness Neuroticism Ambition Grit Risk Attitude Discount Rate Extraversion 0.341*** Agreeableness Neuroticism Ambition 0.428*** 0.250** 0.247** Grit 0.225* 0.449*** *** 0.423*** Risk Attitude 0.187* ** 0.222* Discount Rate Female *** 0.313*** * *** * p<0.05, **p<0.01, ***p<

9 3.2. Probability of a Correct Answer In this section we show the relationship between the probability of a correct answer over time and certain personality traits. More specifically we look at the Big 5 personality traits, Grit and Ambition. We also look at economic preference parameters such as the risk attitude and the time preference of an individual. We split the sample into two groups namely below and above the median of the distribution of each respective trait and preference. The dark grey area always indicates the 95% confidence bounds of the probability of the correct answer over time for the lowest 50 percent of the distribution. The light grey area always indicates the 95% confidence bounds of the probability of the correct answer over time of the highest 50 percent of the distribution Big Five Figure 1 shows the probability of knowing the correct answer over time for the Big Five taxonomy. Panel A shows the probability of knowing the correct answer for the trait openness to experience. It documents that subjects who are more open to experience have a higher chance of knowing the probability to a test question. Panels B to D show the relationship for conscientiousness, extraversion and agreeableness. We do not find an impact on knowing the correct answer over time for these personality traits. Panel E shows the relationship between the time a test-taker thinks about a question and the probability of a correct answer for neuroticism. Subjects who score lower on the neuroticism scale find the correct answer faster than subjects who score higher on the neuroticism scale. 8

10 9

11 Figure 1. The relationship between the Big Five and the probability of knowing the correct answer over time. The grey areas indicate the 95% confidence interval. 10

12 Grit and Ambition Figure 2 shows the relationship between the probability of knowing the correct answer over time split by different types of ambition (Panel A) and grit (Panel B). More ambitious subjects seem to know the answer a bit faster compared to subjects in the lower part of the ambition distribution. However, the difference vanishes when time elapses. We cannot identify an impact of grit on the probability of knowing the correct answer to a test question. 11

13 Figure 2. The relationship between Ambition and Grit and the probability of knowing the correct answer over time. The grey areas indicate the 95% confidence interval. 12

14 Risk and Time Preferences Panel A of Figure 3 shows the relationship between the probability of knowing the correct answer over time and an individual s risk attitude. Subjects who are risk averse are overall less likely to know the correct answer to a test question. The differences is highly significant after the 40 th second. Panel B of Figure 3 shows the relationship between an individual s discount rate and the probability of knowing the correct answer to question over time. More patient subjects have a higher probability of knowing the answer in the first five seconds. The over all probability of knowing the answer remains lower for impatient subjects. However, the difference is not statistically different. Overall, we find a highly significant impact of an individual s risk attitude on the probability of knowing the correct answer to a test question over time. We argue that not only the technology of answering but also the behavior during a test determines the final result. Hence we will analyze the relationship between personality traits, preferences and answering behavior in the following section. 13

15 Figure 3. The relationship between economic preferences and the probability of knowing the correct answer over time. The grey areas indicate the 95% confidence interval. 14

16 3.3. Answering Behavior Table 2 shows the determinants of submitting an answer to a test question. The dependent variable is the time in seconds when a subject submitted the answer to question. All regressions contain question fixed effects. We cluster the standard errors at the individual level to take into account correlations between questions and individuals. The picture that emerges from Table 2 is that besides the treatments only the discount rate plays a crucial role in determining the answering behavior. Subjects who are impatient submit their answer significantly earlier. 15

17 Table 2: Determinants of Submission Time (1) (2) (3) (4) (5) HH (0.439) (0.438) (0.438) (0.439) (0.417) HL 2.879*** 2.888*** 2.884*** 2.880*** 2.862*** (0.483) (0.484) (0.483) (0.483) (0.459) Openness (0.609) (0.669) (0.709) Conscientiousness (0.642) (0.601) (0.734) Extraversion (0.550) (0.601) (0.666) Agreeableness (0.548) (0.518) (0.608) Neuroticism (0.601) (0.661) (0.744) Ambition (0.702) (0.927) (0.977) Grit (0.817) (0.881) (0.957) Risk Attitude (0.668) (0.718) (0.808) Discount Rate ** ** * (0.774) (0.791) (0.825) Constant 22.77*** 22.74*** 22.77*** 22.78*** 22.90*** (1.064) (1.068) (1.075) (1.073) (1.073) Observations 5,521 5,521 5,521 5,521 5,521 R-squared Question FE YES YES YES YES YES Treatment Interactions NO NO NO NO YES Note. The table shows results of a linear panel regression. The dependent variable is the time of submission in seconds. All preference and personality measures are standardized. Standard errors are clustered at the level of the individual (128 clusters). *p<0.05, **p<0.01, ***p<

18 An important question is what determines the submission of a correct answer. More specifically the submission of a correct answer is determined by the speed an individual comes up with the correct answer and the point when an individual decides to submit an answer. Table 3 shows the determinants of a submitting a correct answer. The dependent variable takes the value 1 if the answer to a question was submitted correctly. All regressions contain question fixed effects and the standard errors are clustered at the individual level. The main determinants of submitting a correct answer is an individual s discount rate and the risk attitude. Individuals with a high discount rate have a significantly lower probability of submitting a correct answer independent of the treatment. Moreover, risk lovers have a higher probability of answering correctly. All personality traits do not yield significant results. 17

19 Table 3: Determinants of Correct Submission (1) (2) (3) (4) (5) HH - Treatment (0.0450) (0.0451) (0.0454) (0.0455) (0.0451) HL - Treatment (0.0406) (0.0407) (0.0409) (0.0411) (0.0404) Openness (0.0384) (0.0408) (0.0529) Conscientiousness (0.0386) (0.0386) (0.0488) Extraversion (0.0358) (0.0362) (0.0423) Agreeableness (0.0363) (0.0352) (0.0461) Neuroticism (0.0383) (0.0446) (0.0513) Ambition (0.0411) (0.0506) (0.0613) Grit (0.0439) (0.0529) (0.0585) Risk Attitude ** * (0.0328) (0.0370) (0.0479) Discount Rate * * (0.0413) (0.0407) (0.0492) Constant 0.597*** 0.595*** 0.599*** 0.602*** 0.604*** (0.123) (0.123) (0.123) (0.123) (0.123) Observations 5,741 5,741 5,741 5,741 5,741 Question FE YES YES YES YES YES Treatment Interactions NO NO NO NO YES Note. The table shows the results of a probit regression. The dependent variable takes the value 1 if an answer was correctly submitted. All preference and personality measures are standardized. Standard errors are clustered at the level of the individual (128 clusters). *p<0.05, **p<0.01, ***p<

20 5. Conclusion We conduct a controlled laboratory experiment to examine the mechanisms of personality traits and economic preferences in determining a test result. Our data shows evidence that for instance neurotic people have a lower probability in knowing the correct answer over time. However one of the main determinants for a good test result seems to be an individual s discount rate and her risk preference. More patient individuals have an overall higher probability of knowing the answer during a test and they wait longer until they submit their answer. Risk lovers seem to have a higher probability of knowing the answer at any point in time. 19

21 References Almlund, Mathilde, et al. "Personality Psychology and Economics1." Handbook of the Economics of Education 4.1 (2011). Borghans, Lex, Huub Meijers, and Bas Ter Weel. "The role of noncognitive skills in explaining cognitive test scores." Economic Inquiry 46.1 (2008): Borghans, Lex, Huub Meijers, Benedikt Vogt and Bas Ter Weel. The Economics of Test Taking: The Effect of Pressure on Decision Making and Test Performance Working Paper. Maastricht University (2014). Carpenter, Patricia A., Marcel A. Just, and Peter Shell. What One Intelligence Test Measures: A Theoretical Account of the Processing in the Raven Progressive Matrices Test. Psychological Review, 97(3) (1990): Dohmen, Thomas, et al. "Are risk aversion and impatience related to cognitive ability?." The American Economic Review (2010): Duckworth, Angela L., et al. "Grit: perseverance and passion for long-term goals." Journal of personality and social psychology 92.6 (2007): Duckworth, Angela Lee, et al. "Role of test motivation in intelligence testing." Proceedings of the National Academy of Sciences (2011): Falk A, Dohmen T, Huffman D, Sunde U, Becker A. A validated preference module. Mimeo Fischbacher, Urs z-tree: Zurich Toolbox for Ready-made Economic Experiments. Experimental Economics, 10(2):

22 Goldberg, L. R. An Alternative Description of Personality: The Big Five Factor Structure. Journal of Personality and Social Psychology, 59, 1990, Raven, J. C Advanced Progressive Matrices: Sets I and II. Published by H.K. Lewis & Co. Ltd., London. 21

23 6. Appendix 6.1. Additional Information on the Experimental Design Figure 6.1. Screen shot of the decision making screen for a typical Raven matrix. The matrix is taken from Carpenter et al (1990) to keep the actual Raven matrices confidential Personality Measures Big 5 22

24 Grit and Ambition 23

25 6.3. Preference Measures 24