Neuroscience and Biobehavioral Reviews

Transcription

1 Neuroscience and Biobehavioral Reviews 36 (2012) Contents lists available at SciVerse ScienceDirect Neuroscience and Biobehavioral Reviews jou rnal h omepa ge: Review Moral dilemmas in cognitive neuroscience of moral decision-making: A principled review J.F. Christensen, A. Gomila Human Evolution and Cognition, Associated Unit to the IFISC (CSIC-UIB), Department of Psychology, University of the Balearic Islands, University Campus, Building: Guillem Cifre de Colonya, Palma, Spain a r t i c l e i n f o Article history: Received 31 August 2011 Received in revised form 12 January 2012 Accepted 6 February 2012 Keywords: Moral dilemmas Moral decision-making Moral judgment Moral psychology Neuroethics a b s t r a c t Moral dilemma tasks have been a much appreciated experimental paradigm in empirical studies on moral cognition for decades and have, more recently, also become a preferred paradigm in the field of cognitive neuroscience of moral decision-making. Yet, studies using moral dilemmas suffer from two main shortcomings: they lack methodological homogeneity which impedes reliable comparisons of results across studies, thus making a metaanalysis manifestly impossible; and second, they overlook control of relevant design parameters. In this paper, we review from a principled standpoint the studies that use moral dilemmas to approach the psychology of moral judgment and its neural underpinnings. We present a systematic review of 19 experimental design parameters that can be identified in moral dilemmas. Accordingly, our analysis establishes a methodological basis for the required homogeneity between studies and suggests the consideration of experimental aspects that have not yet received much attention despite their relevance Elsevier Ltd. All rights reserved. Contents 1. Introduction The context Moral dilemma research to date Rationale behind a moral dilemma Dilemma formulation Presentation format Expression style Word framing effects Word number count Participant perspective Situational antecedent Order of presentation Type of question Justifications The experimental participant and her relatedness to the story characters Demographic variables of the participant In/outgroup Kinship/friendship Speciesism Dilemma conceptualization Intentionality Kind of transgression Corresponding author at: University of the Balearic Islands, University Campus, Department of Psychology, Building: Guillem Cifre de Colonya, Palma, Spain. Tel.: ; fax: /$ see front matter 2012 Elsevier Ltd. All rights reserved. doi: /j.neubiorev

2 1250 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) Directness of harm The trade-off Normality of harm Certainty of events Schematic overview of 25 studies in relation to the 19 design parameters Conclusion and discussion Conclusion Future directions: a theoretical framework for working hypotheses Acknowledgements References the use of artificial moral dilemmas to explore our moral psychology is like the use of theoretical or statistical models with different parameters; parameters can be added or subtracted in order to determine which parameters contribute most significantly to the output. (Hauser et al., 2007) 1. Introduction 1.1. The context A morning press release in a daily newspaper informs the wellprotected public of a terrible incident in a foreign war-swept country. Apparently rebel troops assaulted a little village on the search for enemy soldiers and recklessly killed also innocent civilians. Frightened, the village inhabitants had hidden together in small groups so not all had been found and killed. When the enemy had finally left, however, it turned out that one of the women had smothered her baby. She had been trying to keep it quiet so its cries wouldn t give away her group s hiding place to the soldiers, who would have killed them all on the spot. Such horrible stories about real-life scenarios where people are forced to make moral decisions about life and death, flicker over our TV screens every day. Or, they are brought to us by the radio, via the internet when checking our s, or by friends who tell us about what they ve heard. Confronted with this kind of information, we all immediately have feelings about it, and we make judgments of approval and reproach. Some of us may even be asking ourselves, what would I have done in her place? Of course we then happily recognize how lucky we are not having to make such moral decisions involving life and death. But, what if one is taken to a cognitive neuroscience laboratory and asked the same question? Would you...? In the quest for the foundational principles of human moral cognition, cognitive scientists have done exactly this: asked experimental participants to judge such morally dilemmatic situations. It is clear that this experimental set-up does not allow to study real-life-or-death decisions, but this is not the intention here (an analysis of real-life decisions made in dilemmatic situations throughout the turmoils of the 20th century would serve that goal). Conversely, moral judgments of hypothetical real-life moral dilemmas provide the cognitive scientist with valuable insight into the foundational psychological processes that underlie human moral cognition. Thus, considering the example above, human moral judgment generally deems it wrong to kill a baby, but experiments have shown that there are many variables that influence how an individual eventually judges a moral transgression such as this one (smothering the baby). What if the person to sacrifice to save the others was not a baby, but a fellow adult? A foreigner? What if the protagonist (here the mother) would not be killed if the soldiers found them, only the men in the group? Would you smoother your baby if...? And so forth. When compared to other experimental approaches used in empirical studies of moral cognition such as paradigms involving semantic judgments to sentences with moral content (Heekeren et al., 2003), judgments of disgust and indignation to sentences with moral-emotional connotations (Moll et al., 2005), or moral judgments after participation in game tasks such as the Dictator or Ultimatum games (Hofmann and Baumert, 2010; Takezawa et al., 2006) moral dilemmas present a series of advantages: first, they permit the inclusion of many more variables in the formulation than in single sentences only, making possible a more holistic approach. Second, they allow the inclusion of all these variables under a higher level of experimental control, as compared to other approaches: the dilemmas are exactly the same for each individual participant and not subjected to the variability that may occur when different individuals and even actors intervene in the experiment. Third, a skeptic may remark that the extreme nature of some moral dilemmas simply grasps the reader s attention in such brusque manor that subtle variations in the dilemma formulation suffice to trigger distinct moral judgments. However, reality tells a different story. As shown by means of the press release example above, even extreme moral conflicts may be part of all individuals everyday life. Moral dilemmas allow one to elicit these moral conflicts and to thoroughly investigate which parameters our basic moral intuitions respond to and all this, under a high level of experimental control. Consequently, a growing number of authors argue that moral dilemmas such as the famous Crying Baby dilemma above (Greene et al., 2001) offer a valuable tool to study closely which factors trigger the underlying psychological processes that constitute the foundations of human moral cognition (Greene, 2008; Haidt and Graham, 2007; Hauser et al., 2007; Nichols and Knobe, 2007). This approach will ultimately allow us to draw conclusions about real-life moral decision-making which draw on these foundational psychological processes Moral dilemma research to date The past decade has witnessed a blossoming of studies in Moral Psychology and Neuroethics. Following the lead of Damasio and his colleagues (Damasio, 1995) in providing neuroimaging evidence that emotional processing is involved in decision-making, many studies that have focused on moral judgment and ethical decision-making have given rise to models that also link moral judgment to emotion (Greene, 2008; Greene et al., 2002, 2004, 2001; Haidt, 2001; Moll et al., 2002, 2005). Among the many experimental paradigms used in this field of research, moral dilemmas are very popular. The reason, as the initial quote points out, is clear: by means of an adequate moral dilemma design, this methodology allows to systematically explore how distinct parameters modulate our moral judgment (Hauser et al., 2007). However, there has been a lack of principled analysis of the relevant parameters involved in moral dilemma formulation, and research appears to have proceeded in a rather piecemeal fashion. Now, the methodological heterogeneity in the field has reached such a level that the

3 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) obtained evidence is neither necessarily comparable nor replicable across studies. In this paper, we review the variations in experimental paradigms using moral dilemmas, and their criticisms. Our overall conclusion is that, in spite of all criticism, the use of moral dilemmas can be a promising research strategy, when their foundational parameters are accounted for in their complexity. 2. Rationale behind a moral dilemma A moral dilemma is a short story about a situation involving a moral conflict. A moral conflict is a situation in which the subject is pulled in contrary directions by rival moral reasons. It entails the awareness of the incompatibility of two courses of action and their subsequent outcomes. For some ethicists, sui generis moral conflicts are theoretically impossible as they are incompatible with an inclusive moral theory such conflicts would amount to an unacceptable blindspot (Sorensen, 1988). However, from the point of view of Moral Psychology it is obvious that we can and often do experience moral conflicts. Moral conflicts can be of many different types, such as (i) conflicts between personal interests and accepted moral values, (ii) conflicts between different duties, (iii) conflicts between sets of apparently incommensurable values, and even (iv) conflicts stemming from one unique moral principle, as in Sophie s Choice (Styron, 1979). Typically, the scenario that we face in a moral conflict and hence also in a moral dilemma is that both options have important moral reasons to support them. Kohlberg (1964) was the first to use moral dilemmas in Moral Psychology. He was interested in the development of moral reasoning, that is, how reasons given to prefer one choice over another in a conflict situation change with age and degree of moral development. He was sensitive to Piaget s position that children come to distinguish two distinct kinds of moral reasons: one that judges actions according to their material consequences and one that only takes intentions into account. These two attitudes may co-exist at the same age and even in the same child [...]; we have therefore two processes partially overlapping, but of which the second gradually succeeds in dominating the first (Piaget, 1932/1965). Kohlberg contended that moral development culminates when a person holds strong principles, in a Kantian way (Kohlberg, 1964). This approach has been criticized for several reasons such as (i) for over-intellectualizing moral judgment, (ii) for its explicit plea for Kantian ethics over Utilitarianism, and (iii) for overlooking the bias produced by post hoc rationalizations about a choice, as when the reasons given for the choice do not necessarily reflect the actual cause of that choice, but are the fruit of mere post hoc rationalization and the phenomenon of moral dumbfoundedness (Haidt, 2003). Thus, of course, a Kantian agent may also be challenged by a moral dilemma (e.g. in a conflict of duties). In Neuroethics, moral dilemmas were first introduced by (Greene et al., 2001), as a way to develop a paradigm for experimentally induced cognitive conflict in this area (Gomila, 2007), and many studies have followed suit. They found inspiration in Ethics where dilemmas are sometimes used as thought experiments, as intuition-pumps that can reveal conceptual inconsistencies in our moral intuitions. As a matter of fact, the inspirational dilemmas, the Trolley and Bystander dilemmas (Foot, 1967; Thomson, 1976), were originally instrumental in arguing for the inconsistency of Utilitarianism (or Consequentialism, in general) as an ethical theory one which advocates choosing the option that produces the highest welfare for the largest number of individuals involved. In the original Trolley dilemma, a runaway trolley is heading for five railway workers. The driver of the trolley has the option to divert it to another track where only one worker will be killed. Given the situation, the option of killing one instead of five seems justified to most people. According to Foot (1967), though, if consequences were all there was for the morally best option, as Utilitarianism contends, it would also be morally right for a surgeon to kill one patient to save five by transplanting the former s organs to her other patients an option that everybody rejects. Utilitarianism, in conclusion, misses a morally relevant difference between both situations, which has nothing to do with the best overall outcome. Our moral psychology responds to finer differentiations than the mere motivation to obtain the higher pay-off or the lesser evil. In Thomson s version of the trolley dilemma, two scenarios are contrasted again. In the first, the protagonist can change the course of the trolley by pulling a switch which will redirect the trolley on to another track where it will kill only one railway worker instead of five; or, if she considers this to be morally wrong, omit to carry out any action. In the footbridge version of the dilemma, the proposed action to save five workers is different, whilst the plot and the options are the same. This time, the participant has the choice between pushing with his own hands a large person onto the tracks in order to stop the trolley from killing the five. While most people intuitively accept that the switch should be pulled in the first scenario, most people consider that they would not push the large man, even if the consequences are the same: one person is killed to save five. Given that the balance of gains and losses, in all these situations, is the same (one dead to save five), Utilitarianism again cannot account for this fully as an ethical theory: it fails to capture a morally relevant aspect of cognition that distinguishes between the first and the second pairs of scenarios. The shortfall of Utilitarianism has also been discussed on the grounds that it requires giving the same moral weight to the interests of unknown people and future generations, as that given to family and friends. From this it follows that, apparently, Utilitarianism is an impersonal theory of moral right, while our moral psychology is a radically personal one (Williams, 1973). Echoing this criticism of Utilitarianism, Greene et al. have argued that what differentiates the first kind of situation from the second is that the first is an impersonal dilemma, while the second is a personal one. Ironically, however, at the same time they took for granted that most people are utilitarian in their moral judgment (Greene et al., 2004, 2001). We will come back to the Personal Impersonal distinction in Subsection 5.3, directness of harm. For experimental purposes, moral dilemmas are usually variations of the basic trolley/bystander/footbridge scenarios. Each moral dilemma presents a short story about a situation involving risk of harm to one or more individuals. In the last part of a moral dilemma, an alternative course of action is proposed (involving action vs. action omission by the participant) which would spare some of these individuals, but not all. This alternative course typically results in less harm in terms of overall numerical outcome, or in other benefits as compared to the first envisaged outcome. However, it involves committing a moral transgression towards a third party, otherwise not harmed (see the trade-off in Subsection 5.4). Such harm can be of several forms such as physical harm, killing, or social harm as for instance, lying, stealing, unfair behavior, lack of respect (see Subsection 5.2 for an analysis of the different kinds of transgression). For this alternative course to take place, the protagonist of the story has to leap in, carrying out the proposed moral transgression (variations to find here are in the directness of harm variable, see Subsection 5.3), which may lead to intended harm (i.e. the use of harm as a means) or harm as a side effect (i.e. harm as a non-intended side effect, see Subsection 5.1, intentionality ). After reading the dilemma, the experimental participant is asked to judge whether or not the protagonist should carry out the moral transgression. This choice is commonly referred to as moral judgment. Given this general structure of the dilemmas, for the purpose of our analysis, we divide the relevant experimental design

4 1252 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) Fig. 1. In this figure we summarize the variables discussed throughout this paper as being important in moral dilemma design. The figure shows the time course of a single experimental trial presenting a moral dilemma to an experimental participant. It shows the different variables that can be manipulated by the experimenter. The moral judgment of the participant will vary as a function of the weighting of the different variables. The large grey arrow shows the time course of the dilemma as presented to the participant. As she begins to read the dilemma, she obtains information about the preliminary situation of the dilemmatic encounter and is asked to take a perspective to the protagonist of the story. Then, the dilemmatic situation is described. Subsequently, the dilemma formulation will lead the participant to understand the two options of outcome that there are for the dilemma and propose an action of moral transgression to change the first described course of events. With this, the participant sees herself in the situation of having to choose between two sets of victims. The first set would be harmed if the participant chooses not to intervene in the situation and the second set of victims would only be harmed if the participant chooses to carry out the proposed moral transgression. It is very important how the dilemma is formulated which we discuss in Section 3 of this paper. Furthermore, in Section 4 we show how the experimental participant will be influenced by a wealth of variables concerning her relatedness to the other characters in the dilemma. Finally, in Section 5 of this paper we explore the dilemma conceptualization which has to be explicit in the dilemma formulation from the outset. The variables described in all three sections interact in the process that will eventually lead the experimental participant to carry out one moral judgment or another. parameters into three categories: (i) dilemma formulation refers to how the story is presented, what kind of participant response is required and how to trigger answer effects by manipulating variables such as the expression style or the wording of the dilemmas etc. (Section 3); (ii) participant characteristics describes the experimental participant and her relatedness to the characters in the story (Section 4); and (iii) dilemma conceptualization describes the morally relevant elements that characterize the situation as understood by the participant (Section 5) (See Fig. 1 for a schematic overview of the dilemma structure). 3. Dilemma formulation 3.1. Presentation format In behavioral studies, the experimental task can either be computer-based or participants can be asked to complete a simple pen-and-paper questionnaire. A drawback of the pen-and-paper method is that RT assessment is not possible. Besides, an experimental comparison between the pen-and-paper method and the computerized presentation could be of interest, to find out whether it makes a difference to have a questionnaire in front of you, where you see your answers and can go back to read them, or to answer by means of a computer keyboard, without this possibility. In neuroimaging studies the presentation format is usually computerized. Participants read the dilemmas and make their judgments inside the scanner (as, for instance, in Greene et al., 2004 and Greene et al., 2001), or merely read the dilemmas inside the scanner and make their judgment after the neuroimaging session (Berthoz et al., 2002). Recently, web-based computerized presentation has become a popular option due to the advantage of assessing a broader participant pool which is not limited to the most commonly used participant population: undergraduates (Kraut et al., 2004; Nosek et al., 2002). Nevertheless, the obvious disadvantage of web-based administration is that no rigorous control can be made of participants responses to questions such as age, gender, religious affiliation, nationality or even of the operational competence in the language in which the dilemmas are written (Hauser et al., 2007). See also Greenwald et al. (2003, p. 199) for a discussion of limitations of web site data in social cognition tasks. Another aspect of the presentation format is how the task is actually presented on a computer screen. Commonly, dilemmas are displayed on the screen as blocks of text (Greene et al., 2001)

5 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) (though another option was explored by Greene et al. (2008)) who presented the dilemmas as horizontally streaming text (left to right) with a 36 pt. courier font, at approximately 16 characters per second (Greene et al., 2008). As a general recommendation for future neuroimaging studies using moral dilemmas, we suggest using the procedure proposed by Greene et al. (2001) which has been followed also by numerous others (Borg et al., 2006; Moore et al., 2011a,b). It consists of a computer-based set-up, using three text-screens for the presentation of each dilemma. The first two bits of text are presented on the same screen: the first explains the general situation and the second, which is added to the previous screen by button-press, proposes the alternative course of action including the moral transgression. The third screen clears the previous two and consists of the question eliciting the moral judgment of the participant. This clear order allows one to control explicitly for the moment in which the participant is exposed to each piece of information in the dilemma (see also Subsections 3.6 and 3.7 for discussions on the importance of the order of presentation and on the impact of a situational antecedent in the process of judgment formation). The last aspect of the presentation format regards the presentation time of each screen, i.e. whether to leave the task self-paced or to introduce a time limit. We suggest that each screen should have a time limitation. This springs from two considerations. Firstly, the dilemma task is rather long and runs the risk of participant fatigue. Secondly, if the aim is to trigger basic moral intuitions, response time should not be free, as this could result in deliberate moral reasoning. Deliberate, explicit reasoning may mask implicit moral intuitions, as shown by another popular methodology used in social cognition experiments: the Implicit Association Test 1 (IAT) (Greenwald et al., 1998). The authors of the task contend that the IAT effect (prolonged RT towards certain attributes) reveals implicit attitudes that people have which they might not be aware of, or might choose not to reveal if asked explicitly. Thus, this task is an alternative to explicit questionnaire methods because the responses obtained by the latter method may be biased by self-presentation strategies (fakability) or by introspective limits (unawareness) (Schnabel et al., 2008a,b). The possibility of unawareness of one s motives and implicit attitudes has also been assessed in moral dilemma research (see Subsection 3.9 for a discussion of the use of justifications in moral dilemma tasks), and the danger of fakability in moral dilemma tasks remains an unsolved issue. Social desirability is a well-documented phenomenon (Crowne and Marlowe, 1960; Edwards and Horst, 1953; Richman et al., 1999) and is likely to occur also in moral dilemma tasks. Consequently, strategies to prompt fast and more intuitive (or implicit) moral judgments constitute an important challenge for future studies. In addition, we recommend a closer examination of a proposal by Greenwald and colleagues to check participant data individually for its validity before analyzing the whole data set and eliminate inconsistent or suspicious data points. Their suggestion stems from an effort to develop a better scoring key for the IAT and involves taking into account each participant s response latencies (their SD) to determine whether a trial should be included or not. The algorithm 1 Brief description of the IAT: the IAT aims to assess implicit attitudes to a variety of social phenomena by measuring their implicit underlying automatic evaluation. It does this by measuring automatic associations between a bipolar target (for instance, me vs. others ) with a bipolar attribute (for instance, shy vs. sociable ). Thus, participants are instructed to carry out a series of word sorting tasks as fast as possible. The rationale of the task is that faster responses will occur when two highly associated words (for instance, white + pleasant ) require the same response key instead of different ones. Conversely, for less implicitly associated words or concepts (such as perhaps for some people black + pleasant ) a longer response time is expected when these two words require a single response key. involves, for instance, systematically eliminating trials with latencies over a certain time point, and excluding the complete data set of a participant if 10% of trials are under a minimum time point for response latencies (Greenwald et al., 2003); and see also Cvencek et al. (2009) for an example of how the authors detect fakability with statistical methods Expression style Borg et al. (2006) argued that the lack of control of expression styles in dilemma formulation could bias the subsequent moral judgments. From this it follows that, if dilemmas of one dilemma type are formulated with a very colorful style, using floweryemotional adjectives, whereas the dilemmas of the other type are crammed with technical vocabulary and abstract reasoning concepts, then there is a reasonable danger that differences in moral judgment, emotional arousal, and underlying neural activity may be due to the differences in the expression style and not due to the intended conceptual differentiations between dilemmas. In fact, in their 2006 fmri study, these authors found a significant effect of the language used in the dilemma formulation on neural activity. To test their assumption they had created a dramatic (colorful) and a non-dramatic (non-colorful) version of each dilemma. The results showed that while the behavioral effects of this manipulation were modest, there was a significant interaction of Language Morality in brain structures related to emotion processing (anterior cingulate cortex, the posterior orbitofrontal gyrus and the lateral temporal pole). Therefore, the authors conclude that future studies should make an effort to standardize the amount of descriptive and dramatic language in different dilemma categories (Borg et al., 2006). Another relevant piece of evidence in this regard comes from a slightly different methodological set-up. Nichols and Knobe (2007) approached the question, not by studying moral judgments to dilemmas, but by going a step ahead in the chain of events, asking their participants to judge the moral responsibility of agents after the latter have committed a moral transgression. They specifically investigated whether and how peoples attribution of more or less moral responsibility to agents for committed wrongdoing could be manipulated. They found that peoples attributions of moral responsibility varied depending on how the question was formulated. Thus, following abstract theoretical formulations, people did not consider the agent responsible for her deeds, especially when they thought the agent had no alternative option, or if the moral transgression itself was viewed as predetermined by previous actions or by natural laws. Conversely, emotionally salient (dramatic) formulations triggered explicit attribution of moral responsibility in exactly the same, predetermined situations. Therefore, these authors also conclude that the moral intuitions of a person are very likely to be influenced by the affective wording with which a piece of information is presented in a moral dilemma (Nichols and Knobe, 2007). These results suggest that when dilemmas are translated into other languages, it is advisable to control for or to standardize the expressive style and vocabulary in those languages. Story writing is a complicated matter and most people are not necessarily trained in literary translation, so some care is needed here Word framing effects Word framing effects have been shown to affect decisionmaking (Tversky and Kahneman, 1981). Classical work demonstrates that under certain circumstances, experimental participants violate a very basic principle of rationality: the principle of invariance. This principle asserts that one s choices ought to depend on the situation itself, not the way it is described. However, research

6 1254 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) has shown that people have different preferences over exactly equivalent situations because of the way they are described. For instance, people prefer a situation in which half a population is saved to one in which half the population dies in an epidemy; people also prefer a cash discount over a credit card surcharge (Thaler, 2008). See more on this issue also in Mikhail (2007) in his section on structural descriptions (Mikhail, 2007, p. 145). In research using moral dilemmas, Petrinovich et al. (1993), Petrinovich and O Neill, 1996 stressed the danger of framing effects due to use of different vocabulary in the dilemmas. For instance, different moral judgments were elicited depending on whether the outcome of the action was expressed using the word kill, as compared to the word save (Petrinovich and O Neill, 1996; Petrinovich et al., 1993). This is an aspect not yet acknowledged in neuroethics. However, it is an important issue also conceptually related to the discussion about the action vs. omission bias (see Subsection 5.1 on intentionality), which designates the different moral importance people attribute to ways of describing the agent s implication in the course of action in the scenario, even if the general outcome is the same. Finally, the framing effects also raise the general question of moral dilemmas being textual stimuli which inevitably trigger processing in brain regions involved in language processing a fact that could possibly lead to confounds regarding the origin and cause of neural activation. In fact, a skeptical argument against neuroimaging studies of moral judgment has been leveled on the grounds of a metaanalysis by Ferstl and colleagues, who pointed out that the neural underpinnings of moral judgment are suspiciously similar to those involved in different levels of language and text comprehension. 2 This challenges the assumption of specificity of these regions for moral judgment formation and for other processes associated to social cognition, such as Theory of Mind (ToM) (Ferstl et al., 2008). Similarly, skepticism has been voiced specifically regarding the frontal regions of activation found in studies on moral judgment with text stimuli. In this case, the alternative explanation offered proposes that these activations could be due to different levels of task demands associated with the process of text comprehension: as task demands go up due to the need for higher levels of text comprehension, neural activity is also enhanced (Hashimoto and Sakai, 2002; Love et al., 2006; Peelle et al., 2004). However, both these criticisms overlook two central points. First, they ignore the fact that studies of moral judgment using moral dilemmas primarily aim to contrast judgments to two or more types of dilemmas. Hence, if the formulations of different dilemma types are held constant, any confounding activity will be cancelled out because any problematic component is shared across conditions. Second, studies using pictures instead of text stimuli in ToM and moral judgment tasks report similar patterns 2 Studies on moral judgment with moral dilemmas have repeatedly reported activations in the cingulate cortex, the dorsolateral and medial prefrontal cortex (mpfc and DLPFC) and the temporo-parietal-junction (TPJ) (Greene and Haidt, 2002; Greene et al., 2004; Moll and Schulkin, 2009; Young and Saxe, 2009a). However, a different set of studies has shown that these areas are all also specifically implied in various levels of text comprehension. For instance, the TPJ is found active in the process of comprehension of ambiguous text, while comprehension of coherent text activates PCC/Precuneus, DLPFC and mpfc. Metaphor comprehension is associated to activation in the anterior temporal regions, lateral PFC and TPJ. Furthermore, there is a proposal suggesting that the temporal poles are not only implied in memory and episodic memory processes as classically assumed, but also specifically in integrating language into a coherent representation, by associating syntactic, semantic and episodic sources of information. In particular, it appears that the posterior portion of the temporal gyrus is implied in a coherence analysis together with the TPJ, while the mid portion is activated for basic language processing and the anterior and posterior regions ultimately integrate the multimodal information into a coherent representation. of neural activation as studies that use text (Cikara et al., 2010; Ciaramidaro et al., 2007). Of course, it remains an endeavor for future testing to fine-tune our knowledge of the extent to which some regions of neural activation are proper to language processing, or shared, with social or moral cognition. However, the involvement of a brain area in multiple functions is not an exception but the rule (Gomila and Calvo, 2010), and it appears evident that some processes of language comprehension are shared with other processes of moral dilemma comprehension. Nonetheless, we agree that it is crucial to make sure that linguistic components (syntactic complexity, semantic ambiguity, text coherence, expression style, etc.) are carefully controlled for Word number count In 2001, Greene and colleagues elaborated a dilemma set which has been repeatedly used in studies on moral judgment. However, this dilemma set included dilemmas that were very different in terms of word number. Especially unfortunate was that their two dilemma types had rather divergent mean word numbers (Borg et al., 2006; Moore et al., 2008). This is a particularly relevant issue for fmri studies, where the Blood oxygen level dependent (BOLD) response has a particular timing. It is about 46 s long before it returns to baseline, where it should be at the start of the subsequent trial. For this reason, ideally, trial length (here, word number) should be kept constant in terms of reading time between experimental conditions in order to have approximately the same BOLD response timing in each trial. Furthermore, as studies using moral dilemmas imply a lot of reading, keeping the dilemmas as short as possible prevents participant fatigue Participant perspective The formulation of the moral dilemma can implicitly or explicitly impose a perspective on the experimental participant, likely to affect her judgment. In some studies, the experimental participant is either asked to take the perspective of a protagonist: you are standing on a footbridge over trail tracks as... X... happens whereas in others, the dilemma is described in the third person: David is standing on a footbridge over trail tracks as... X... happens. Participants answers were shown to differ, depending on the participant perspective in the dilemma (Royzman and Baron, 2002). At the neural level, the activation that correlates with the experience of agency is different from the activation observed when merely observing the same action committed by another person (Farrer and Frith, 2002). If we take moral emotions into account (Tangney et al., 2007), in the case of judging one s own morally right or wrong actions, emotions like guilt or shame or even fear of negative social evaluation will guide our moral judgment (Berthoz et al., 2006; Finger et al., 2006; Takahashi et al., 2004). Conversely, if we are judging another person s action, emotions such as indignation and anger will prevail in moral judgment (Moll et al., 2008a,b; Shaver, 1985). Consequently, especially neural activity in emotion processing regions is likely to differ between the two perspectives. A recent study provides support for this view, showing a differential neural signature when participants judged their own and others moral and immoral actions (Zahn et al., 2009). Accordingly, the authors discuss that this difference is mainly related to the differential moral emotions that are elicited by one s own actions (shame, guilt, pride; enhanced neural activity: PFC and anterior temporal lobe activations) and those of others (indignation, anger, praise; enhanced neural activity: lateral orbitofrontal cortex, insula and DLPFC activations).

7 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) Given that the underlying neural, cognitive and emotional mechanisms are not the same, we conclude that the two participant perspective formats can not be considered equivalent for moral dilemma formulation. Hence, a challenge to dilemma formulation is to take into consideration that human moral psychology responds differently to the observation and commission of moral harm, and to the individual s perspective on it Situational antecedent The antecedent or initial circumstances variable refers to any description of actions or situations that have led to the dilemma situation. It might be a reference to the previous physical or interpersonal situation or a concrete allusion to previous actions by the protagonist or the future victim (Mikhail, 2007). For instance, a situational antecedent could reveal that the proposed moral transgression would, as a matter of fact, be an act of self-defense (Nichols and Mallon, 2006). Such a situation can lead a participant to be more inclined to consenting harm, than if such a previous encounter between agent and victim had not been described. In real-life scenarios, a situational antecedent will almost always have preceded a problematic situation and this will influence our moral judgment. This fact makes this variable an interesting means of triggering divergent moral judgments for the same action, depending on the kind of antecedent (Bjorklund, 2003; Cushman, 2008; Royzman and Baron, 2002; Young and Saxe, 2009b) Order of presentation Early research in Moral Psychology revealed that the presentation order of intentions and consequences in a situational antecedent influences moral judgment (Karniol, 1978; Keasey, 1978). In general, this variable, as the former two (word framing effects and situational antecedent descriptions), concerns the pragmatic understanding of narratives, and the well-known phenomenon of implicature (Grice, 1975): people understand more than is said, even if care is taken to make the story fully explicit. Thus, how the story is deployed affects how it is understood and judged. To deal with this aspect of narrative understanding, we suggest maintaining the order of presentation of all relevant information constant throughout all the dilemmas of a set. However, an interesting experimental manipulation is also to counterbalance this presentation order to explore the impact of such a manipulation on moral judgment. This procedure has been largely overlooked in Neuroethics to our knowledge with only one, very remarkable exception. Young and Saxe (2008) showed how the order of presentation of the elements of a dilemma altered the underlying neural activity. Specifically, precuneus and the temporo-parietal-junction showed an enhanced pattern of activation when the agent s belief about the moral transgression was presented at the beginning of the dilemma, as compared to when it was presented later in the narrative. The authors suggest that this piece of evidence underlines the relevance of order of presentation in triggering differential moral intuitions. Consequently, it suggests a wealth of possibilities for future studies to test specifically how different kinds of foreshadow and background knowledge, on the one hand 3.6), and their order of presentation to the experimental participant, on the other (this subsection), may trigger distinct moral intuitions that lead to differential moral judgments Type of question In some studies the participant is asked to rate the appropriateness of the action, mostly in an appropriate inappropriate dichotomy (Franklin et al., 2009; Greene et al., 2004, 2001; Moll et al., 2008a,b; Valdesolo and DeSteno, 2006; Waldmann and Dieterich, 2007). In others, the participant states whether she would choose to carry out the depicted action in a yes no dichotomy (would you...?) (Koenigs et al., 2007), which may be preceded by another dichotomic question asking whether the action is right or wrong (is it right to...?) (Bjorklund, 2003; Borg et al., 2006; Fumagalli et al., 2009; Nichols and Mallon, 2006). Another alternative is to ask the participants to indicate their judgment in a permissible forbidden scale (Cushman, 2008; Cushman et al., 2006). However, beyond framing effects lurking here (see Subsection 3.3), it is not the same to ask whether an action is permissible or appropriate. The first term relates to normative thinking about the legal permissibility of the action (especially, as its counterpart is forbidden), while appropriate suggests whether the participant finds the action obligatory to the situation (where obligation implies permission, but not the other way around). Similarly, the right wrong dichotomy hints very much towards the legal permissibility of the action. On the other hand, the would you...? question does not allow the distinction between the course of action and its normative status (one could decide to do what she takes to be wrong). Considering this panorama, a promising avenue of research is to collect systematic evidence about moral judgments and their neural underpinnings from one single dilemma set with different question formulations. This strategy has been followed by at least three studies so far, yet with slightly diverging conclusions. Cushman (2008) showed that when asked for the wrongness or permissibility of a moral transgression, participants appeared to rely on mental state information about the agent only, such as in the classical example was it wrong of Peter to drive although he was drunk? Conversely, for the assignation of punishment and blame, both mental state and the causal link between the agent and the harmful consequences appear important, such as in should Peter be punished for driving although he was drunk and ran over a girl on his way home? The authors argue that their findings show that two distinct psychological processes underlie these two types of judgment: the first begins with the harmful action and analyzes the mental states behind that action, then judges whether what the agent did was right or wrong (driving although he knew that he was drunk). In absence of harmful consequences the process stops here and no punishment or major blame is assigned. However, if consequences do occur, the causal chain that lead to the action including the mental state becomes more relevant and blame and/or punishment are assigned severely (in line with the discussion of moral responsibility by Nichols and Knobe (2007) outlined in Subsection 3.2; see also the analysis of in Subsection 5.1). The fact that Joe ran over a girl is blameworthy enough, yet the fact that he knew he was drunk when he drove off makes the assignation of blame and punishment even more severe. Also O Hara et al. (2010) argue that different questions may entail differential moral judgments which would ultimately interfere with the objective to develop a unified theory of moral judgment (O Hara et al., 2010). They address this concern empirically by using four different question labels: wrong, inappropriate, forbidden and blameworthy and found that people judged moral transgressions more severely when the words wrong or inappropriate were part of the formulation, than when the words forbidden or blameworthy were used. The authors do not make any specific suggestions about which factors in the formulation may be triggering these differences. Furthermore, surprisingly, in contrast to Cushman (2008), they conclude that the relatively small effect sizes in their study demonstrate that the question formats are equivalent and that consequently results across studies with different question formats can legitimately be compared. This conclusion is unjustified for several reasons: (i) the findings by Cushman (2008) outlined above; (ii) the fact that the small

8 1256 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) effect sizes of their study might be due to a rather heterogeneous sample (web-based assessment); and (iii) also the following study that suggests otherwise. Borg et al. (2006) found different behavioral effects following the question Is it wrong to...? and Would you? (Borg et al., 2006). As compared to judgments of nonmoral scenarios, the question Would you...? resulted in faster RT, while the question Is it wrong to...? did not show any differences in RT as compared to the nonmoral condition. Again, no specific analysis is provided as to which conceptual or moral intuition distinctions may underlie these differences. The authors merely suggest that in view of this finding, it is likely that deciding what to do is processed differently than deciding whether an action is wrong or not. For the scope of the present review it will suffice to conclude that these three studies show that there is potential for promising future testing ahead to determine more reliably which moral intuitions psychological processes are specifically triggered by which kind of linguistic label in the question formulation. As to the question format dichotomic or Likert scales there appears to be a tendency to use dichotomic answer formats across studies. See Table 1 for the different questions and response formats used throughout studies Justifications Despite criticisms of Kohlberg s method of asking for reasons for the judgment after it was made, evidence suggests that in some circumstances this procedure may be an interesting way to study whether the participants are aware of the variables that make them judge a dilemmatic scenario in one way or another (Cushman et al., 2006). It is the contrast between apparently similar pairs of situations which elicits divergent moral judgments that makes such a justification task suitable for some research hypotheses. Moral dumbfoundedness is not the default situation but rather an exception, given that reflexive moral reasoning can influence moral judgment. Dissociations between reflexive reasons and intuitive ones then, when they occur, are also a relevant piece of evidence that any model has to account for. In consequence, we find this procedure a very elegant way to assess the participants awareness of their motivations for making one or another judgment. Yet, we suggest, firstly, to make participants elaborate these justifications only at the end of the whole experiment (and not after each dilemma) and, secondly, to wait until the end of the experiment before telling them that they will have to give such justifications for their judgments, in order to avoid a possible conflict between the judgment the participant wishes to make and the anticipation of a failure to provide a satisfactory justification for it. 4. The experimental participant and her relatedness to the story characters 4.1. Demographic variables of the participant Evidence from studies on the influence of participant characteristics on moral judgment is still controversial. However, there is data to support the idea that they can and actually do influence moral judgment to some extent. The crucial demographic variables of the experimental participant are, primarily, ethnic and cultural background, socio-economical status, educational background (Hauser et al., 2007), age (Wang, 1996), gender (Fumagalli et al., 2009, 2010), political orientation (Graham et al., 2009; Haidt and Graham, 2007), level of religiosity (Hauser et al., 2007), thinking style (Lombrozo, 2009), need for cognition (Bartels, 2008; Cacioppo et al., 1984) and sensitivity to reward and punishment (Moore et al., 2011a,b). Although these authors do not always report differences in moral judgment as a function of these variables, often they suggest that failing to find such differences could be due to the homogeneity of their experimental groups. O Neill and Petrinovich (1998) propose that the effects of ethnic background or nationality on moral judgment should be studied in the respective countries of the nationalities to be compared, and not just selecting two nationalities that happen to live in the same country. In their 1998 study on cross-cultural differences in moral judgment they elegantly confirmed such effect of ethnic and nationality by comparing an U.S. sample with a Taiwanese sample in their respective countries (O Neill and Petrinovich, 1998) In/outgroup The in/out group variable (Cikara et al., 2010; O Neill and Petrinovich, 1998; Petrinovich et al., 1993) indicates whether the possible victims of the moral transgression belong to the participant s social group. Does he or she have the same ethnic background, nationality, gender, age or socio-economical status? In spite of the importance of this variable, only few studies on moral judgment successfully make the explicit effort to rule out possible answer effects specifically due to in/out group differences (Borg et al., 2006). Others suggest merely controlling for this variable, using anonymous agents (Hauser et al., 2007). However, one might object that in real-life scenarios, agents are never anonymous, and that it is very important to account for social biases in an inclusive theory of moral judgment: social categorization always occurs (Hewstone et al., 2002; Nelson, 2006; Rabbie, 1981). In the neuroimaging literature, Cikara et al. (2010) showed how the acceptability of causing harm to one to save many can be altered by manipulating whether the individual to be sacrificed belongs to the protagonist s social group or not. Participants were more willing to sacrifice individuals of extreme outgroups (such as the homeless) to save ingroup members ( good Americans ), than fellow ingroup targets. This study shows that the in/out group variable is crucial also for moral dilemma design due to the psychological phenomenon of social categorization (Cikara et al., 2010). In fact, Cikara et al. based their study specifically on the assumption that social categorization is due to stereotype content information in the stimuli. Given this hypothesis, they aimed to empirically test the Stereotype Content Model (Fiske et al., 2007, 2002). This model contends that individuals classify other individuals into four basic categories in a two-dimensional space, where the two dimensions are warmth and competence (high vs. low warmth, and high vs. lowcompetence individuals). According to this model, which emotions the observer will have when something good or bad happens to the individual in the story and which actions she will be willing to engage in for that individual will depend on the grouping into one of the resulting four categories. For instance, if it is a high-warmth and low-competence individual (elderly, children), harm will result in feelings of pity in the observer which will subsequently lead to prosocial helping behaviors. Conversely, if it is a low warmth and low-competence individual (homeless), harm towards this individual is more likely to provoke feelings of disgust which will make the observer withdraw, and so forth. Consequently, as this initial evidence with dilemmatic picture stimuli by Cikara et al. (2010) demonstrates, as with moral dilemma tasks with text stimuli, the in-/outgroup variable shows promise for experimental manipulation because it reflects a general bias of the human mind. Apart from controlling explicitly for this variable in order to target distinct moral judgments as a function of these variables, we propose, furthermore, to include a measure of prejudice level in the experimental procedure as a means of experimental control. Either in the format of a questionnaire (Phillips and Ziller, 1997), or, by

9 Table 1 Summary of selected studies that use moral dilemmas for moral judgment tasks. Sorted by the variables the authors have controlled for. Methodology Variables Study Type of question * 3.8) Presentation format ** 3.1) Type of study: behavioral/ neuroscientific/ lesion *** Participant perspective **** 3.5) Intentionality 5.1) Situational of the Variables antecedent victim (Subsectiotions 4.2 (Subsec- 3.6) and 4.3) Variables of the participant 4.1) Kin- /friendship variable 4.3) Type of harm/ transgression 5.2) Numerical outcome/ trade-off 5.4) Directness Certainty of harm of events (Subsectiotion (Subsec- 5.3) 5.6) Normality of dilemma 5.5) In- /outgroup 4.2) Dilemma pool by Greene et al. Greene et al. 1 C N 1st v v v v v v v v v v (2001) Greene et al. 1 C N 1st v v v v v v v v v v (2004) Valdesolo 1 C B 1st v v v and DeSteno (2006) Killgore 1 C B 1st v v v v v v v v v v et al. (2007) Koenigs 6 C L 1st v v v v v v v v v et al. (2007) Greene et al. 1 C B 1st v v v v v v v v v (2008) Fumagalli C B 1st v v v v v v v v v v et al. (2009) Fumagalli C N 1st v v v v v v v v v v et al. (2010) Justification part of study Royzman 7 C B v v v and Baron (2002) Bjorklund PP B 1st v v v v v v v (2003) Cushman 3 W B 3rd v v v et al. (2006) Hauser et al W B 3rd v v v v (2007) Greene et al. 1 C B 3rd v v v v (2009) Lanteri et al. 7 PP B 3rd (2008) Lombrozo 3 PP B 3rd (2009) Studies with lesions or pathologies Koenigs 6 C L 1st v v v v v v v v et al. (2007) Ciaramidaro 1 C L 1st v v v v v v v et al. (2007) Franklin 1 C L 1st v v v v v v v v v et al. (2009) Studies with additional experimental manipulation Bjorklund PP B 1st v v v v v v (2003) Valdesolo and DeSteno (2006) 1 C B 1st v v v Self sacrifice controlled? Word number count 3.4) Expression style 3.2) J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012)

10 Table 1 (Continued) Methodology Study Type of question * 3.8) Presentation format ** 3.1) Type of study: behavioral/ neuroscientific/ lesion *** Participant perspective **** 3.5) Variables Intentionality 5.1) Situational of the Variables antecedent victim (Subsectiotions 4.2 (Subsec- 3.6) and 4.3) Variables of the participant 4.1) Kin- /friendship variable 4.3) Type of harm/ transgression 5.2) Numerical outcome/ trade-off 5.4) Directness Certainty of harm of events (Subsectiotion (Subsec- 5.3) 5.6) Killgore 1 C B 1st v v v v v v v v v et al. (2007) Young and 3 C N 3rd Koenigs (2007) Greene et al. 1 C B 1st v v v v v v v (2008) Young and 6 C N 3rd Saxe (2008) Cikara et al. 8 C N 3rd (2010) Studies taking individual differences into account Bartels 6 PP B 1st v v v v v v v v (2008) Berthoz 9 C N v v v et al. (2006) Borg et al. 6 C N 1st v (2006) Killgore 1 C B 1st v v v v v v v v v v et al. (2007) Waldmann 10 PP B 3rd v v v v and Dieterich (2007) Hauser et al W B 3rd v v v (2007) Moore et al. 1 C B 1st v v v v v v (2008) Fumagalli C B 1st v v v v v v v v v v et al. (2009) Fumagalli C N 1st v v v v v v v v v v et al. (2010) % explicitly controlled ( ) % controlled ( not marked ) % unsystematic (v) v = this variable varies unsystematically in the study. = explicitly controlled for. A blank box means that the variable was not explicitly controlled for but that it didn t vary unsystematically or that it simply was not relevant for the objectives of the study. The number captions after the variable names refer to the respective sections in the paper. * 1 = appropriateness of the action; 2 = is it right to...?; 3 = scale forbidden-permissible; 4 = open response format; 5 = agreement/disagreement; 6 = would you...?; 7 = which option is more wrong, morally?; 8 = to what extent is the action morally acceptable?; 9 = post experiment, 10 = should the protagonist of the story take action or not? (six-point Likert scale: definitely not, definitely yes). ** C = computerized task; PP = pen and paper task; W = web-based task. *** B = behavioral study; N = neuroimaging study; L = lesion study. **** 1 = 1st person perspective; 3 = 3rd person perspective. Normality of dilemma 5.5) In- /outgroup 4.2) Self sacrifice controlled? Word number count 3.4) Expression style 3.2) 1258 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012)

11 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) using the IAT to probe for implicit attitudes of prejudice (Greenwald et al., 1998; Sabin et al., 2009) Kinship/friendship The kinship friendship variable is related to the former and is grounded in our evolutionary history. Whereas the in/out group variable relates to social categories, the present variable states that harm directed towards family members, friends, or oneself, has a stronger impact than harm directed towards strangers (Petrinovich et al., 1993). A similar effect occurs if harm is directed towards someone we generally like, or towards someone we dislike (Miller and Bersoff, 1998). Consequently, many studies only include harm directed towards strangers in their dilemma formulation (Hauser et al., 2007; Pizarro et al., 2003; Royzman and Baron, 2002; Valdesolo and DeSteno, 2006). However, including friends or family members as part of the dilemma situation and putting the protagonist s own life at stake represent another experimental manipulation to account for in the analysis. The distinct moral judgments triggered in this way, are explained by the implausible impersonality of Utilitarianism. This variable can be accounted for as the self other beneficial distinction in moral dilemma classification Speciesism The speciesism variable refers to the variation of the species towards which harm is directed. Petrinovich et al. (1993) discuss that drowning a dog to save five human individuals might seem less morally questionable to an experimental participant than drowning an innocent fellow human for the same objective. In some studies on moral judgment with moral dilemmas, the species variable varies unsystematically (Ciaramidaro et al., 2007; Koenigs et al., 2007; Royzman and Baron, 2002), leading to uncontrolled effects in the analysis. In most studies, though, this variable is not controlled for, but at least the species do not vary either within or between the dilemmas (Killgore et al., 2007; Nichols and Mallon, 2006; Valdesolo and DeSteno, 2006). The latter is probably a recommendable strategy if one does not want to study specifically how speciesism influences moral judgment. 5. Dilemma conceptualization Once the more methodological parameters listed above are held constant, conceptual manipulations lead to the design of distinct moral dilemma types that test specific theoretical assumptions about human moral cognition. This section covers these key determinants of moral judgment discussed in Ethics, Neuroethics and Moral Philosophy. So far, empirical evidence is available for (i) the intentionality of the action, (ii) the kind of transgression depicted in the dilemma that leads to the harm of the third party, (iii) the directness with which it is inflicted, (iv) the trade-off of goods and wrongs for each alternative, (v) the degree of normality of the inflicted harm, and (vi) the certainty of events Intentionality Following the work of Piaget (1932/1965), it is well-known that our moral psychology is sensitive not only to the consequences of an action, but also to the intentions behind it (Piaget, 1932/1965). Accidental (or unintended) harm is not considered morally wrong, while intentional harm is; but also inevitable harm although intentional, but incurred as a side effect is at times considered acceptable by experimental participants, as is intended harm if it is to produce a further benefit to the recipient. In general, harm that is the outcome of an intention to do good may be judged acceptable (Turiel et al., 1987; Weiner and Peter, 1973; Zelazo et al., 1996). Thus, the pain a doctor may cause to a patient in her effort to cure her (as in chemotherapy, for instance), is deemed morally acceptable, even if the effort is not successful, while the same harm may be judged a moral transgression if it is originated in the mind of a psychopath. What s more, intention is all that matters in cases of failed attempts of harm (Young and Saxe, 2009b). Moral and Legal theory have tried to work out concepts and distinctions to grasp these qualifications and intuitions. Some of the best well-known doctrines were coined by Thomas Aquinas, such as the Doctrine of Doing and Allowing (DDA) (Quinn, 1989). It states that it takes more to justify harm derived from action than from inaction, as in the case of active vs. passive euthanasia, because such a difference makes a moral difference. People tend to consent passive Euthanasia (switching off the vital equipment) but not the active version (for instance, give the patient a lethal injection). This distinction was also acknowledged in the study by Waldmann and Dieterich (2007), which showed that participants are more sensitive to the consequences of action than to the consequences of inaction (agent-patient dichotomy) an effect also called omission bias (Ritov, 1990). Waldmann and Dieterich (2007) proposed an agent-patient dichotomy, according to which action omission would cause an agent to die, whereas action commission would result in an innocent person s death (patient), who otherwise would not have died (Waldmann and Dieterich, 2007). The participants in this study were sensitive to this distinction, responding according to the principles of the DDA. The authors called this phenomenon intervention myopia. Similarly, Moore et al. (2008) alluded to the same principles when they introduced the avoidable inevitable variable in their dilemmas. On the other hand, the Doctrine of the Double Effect (DDE) (Foot, 1967) is the next landmark to consider. In this rationale harm is justified (or forgiven to a certain extend) if it is a side effect that results as an inevitable consequence of an intentional action carried out for a better overall outcome, even if that side effect was foreseen and discounted. No wonder there is a lack of consensus in this regard, many people reject such doctrines (Foot s use of the trolley dilemma also targeted DDE). However, empirical evidence suggests that they actually may be capturing intuitions of our moral psychology (Turiel et al., 1987; Weiner and Peter, 1973; Zelazo et al., 1996). Thus, Hauser et al. (2007) contended that DDE is part of our moral competence, pushing intentionality center-stage in the study of moral judgment. In this approach, the differences in moral judgment in the bystander vs. footbridge versions, beyond directness of harm, are determined by whether the harm is produced intentionally or as a non-desired side effect. Participants choose action omission when the moral transgression implies harm as a means but choose action when the harm derived from the moral transgression is described as a non-intended side effect (Cushman et al., 2006; Hauser et al., 2007). Finally, the importance of intentionality in moral dilemma formulation is also highlighted by the fact that the presence of such formulations in the text have been found to enhance neural activity in a moral judgment and Theory of Mind (ToM) related brain region, the Right Temporo-Parietal-Junction (RTPJ) (Saxe and Powell, 2006). Thus, this brain region specifically responds to the inclusion of intention information in the formulation, as compared to when only other social information is included. In summary, research with moral dilemmas bridges two interesting research questions related to the behavioral and neural effects triggered by different dimensions of intentionality: (i) whether intention backs the production of harm (intentional/instrumental vs. accidental/incidental), and (ii) whether harm occurs by doing or by allowing (action omission vs. action commission). Furthermore, here the dimensions of the dilemma outcome

12 1260 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) and the relatedness of the story characters also become relevant. These dimensions are avoidable harm or inevitable harm, on the one hand, and dimensions of the dilemma outcome self-beneficial or other beneficial, on the other Kind of transgression The type of transgression described in a dilemma is a relevant variable that has been found to account for systematic variations in subsequent moral judgments (Burnstein et al., 1994). In research with moral dilemmas, the harm depicted in the dilemma is usually intended to be specifically moral, while conventional harm is better avoided. This grasps the rationale behind the use of moral dilemmas given that, without moral harm, there is no moral dilemma. However, principled accounts of the kinds of moral transgressions beyond the basic distinction between moral and conventional harm (Turiel, 1983) remain scarce. The Domain Theory of Moral Development (Turiel, 1983) contends that the human mind responds to two main types of harm: morally questionable actions that imply physically harmful consequences for a victim, and, violations of social conventions that do not have observable consequences, other than the transgression of socially or culturally adopted norms or rules. Research has mainly focused on harming others as the core example of a moral transgression, but while it is surely not the only one, it is not easy to specify how many kinds can be differentiated. Regarding different types of specifically moral harm Shweder et al. (1997), for instance, proposed three basic moral domains (autonomy, community and divinity) and argued that harm towards these entities trigger differentiable moral judgments. Haidt and Graham (2007) fine-tune this distinction further in their proposal of the Five Foundations of Morality (Haidt and Graham, 2007). The five domains or moral intuitions (Haidt, 2001) are harm/care, fairness/reciprocity, ingroup loyalty, authority/respect and purity/sanctity (self dignity). According to this view, throughout evolutionary history, human moral cognition has become hard-wired to respond to transgressions of any of these five dimensions. Of course, individuals may vary as to how these intuitions have behavioral effects, depending on enculturation and social education. Nonetheless, Haidt and Graham suggest that these five domains constitute universals of the human moral mind and that harm directed towards any of them will elicit moral indignation and condemnation. To test this specifically in studies using moral dilemmas, we suggest to approach the Five Foundations Theory as a dimensional space and vary the ingredients of each possible harm domain in a controlled manner. See the conclusion of the present paper for details of this proposal. However, whichever theory about the kinds of human moral intuitions turns out to be correct, for moral dilemma design it is simply important to keep in mind that the type of transgression matters. Thus, considering the theoretical panorama of possible harmful actions that can be formulated in moral dilemmas, it is a challenge for future testing to ascertain exactly to what extent our moral judgment is mediated by variations in these different harm domains Directness of harm The first conceptual distinction that was made regarding the directness of harm variable was the personal impersonal distinction by Greene et al. (2001, 2004). This consideration regarding the physical proximity between the agent and the produced harm was based on the Bystander/Footbridge versions of the trolley dilemma (Thomson, 1976). Personal moral dilemmas were defined as those that involve bodily harm, deflect an existing threat onto a different party and befall a particular person or member of a particular group of people (Greene et al., 2001). However, this distinction was criticized for not being clear enough to thoroughly define a distinct feature of human moral cognition (Mikhail, 2007), and Greene himself later stressed the fact that this distinction was only a preliminary proposal of classification (Greene, 2009). It is noteworthy, however, that despite criticisms, there are studies that have elegantly shown how participants are sensitive to manipulations of this variable (Cushman et al., 2006; Pizarro et al., 2003). However, more recently, Moore et al. (2008) reformulated the personal impersonal distinction based on a preliminary definition by Royzman and Baron (2002), as a function of the personal distance with which the harm is carried out, and found participants sensitive to this differentiation (Moore et al., 2008). As a consequence, Greene et al. (2009) introduced a re-shaping of their dilemmas in this respect and labeled the resultant variable personal force as a specific experimental factor. This variable now additionally reflects to what extent the agent s muscles and whole body are implied in the execution of the proposed harm (Moore et al., 2011a,b) The trade-off The trade-off variable in the dilemma formulation refers to the balance of gains and losses that each option produces (Cushman et al., 2006; Nichols and Mallon, 2006; Waldmann and Dieterich, 2007). It concerns how beneficial the alternative outcome has to be in order to justify committing (or passively consenting to the execution of) a moral transgression. In other words, how bad does the situation have to be for an individual to agree to the alternative harm situation taking place? In the standard dilemmas, the tradeoff is one-to-five deaths, but it could be interesting to investigate if there is a turning point (two to one?), were the judgments change. In the same vein, this variable tests how participants respond to the principles of the philosophical Doctrine of the Double Effect (DDE) or to the omission bias, reflected in the Doctrine of Doing and Allowing (DDA). It may become more complex when the set of gains/losses involved are not easily measurable or comparable (in the case of incommensurable goods) Normality of harm Depicted dilemma situations may vary in their degree of normality to a particular subject population. A combat situation might be familiar to a soldier, but not to an undergraduate psychology student. Similarly, an abortion is likely to be more a part of a young Western girl s world than the suffocation of a newborn baby to stop it crying in a situation of war where enemy soldiers are approaching your house (Bjorklund, 2003; Borg et al., 2006; Greene et al., 2009; O Neill and Petrinovich, 1998). One solution to this bias is to ask the participants to rate how normal the depicted dilemma situation appears to them (after judging it) and to use this information as a covariate in the data analysis as did Greene et al. (2009). Another possible solution is to include an allusion to this problem in the instruction to the participants. O Neill and Petrinovich (1998) and Greene et al. (2009) included a short clause explaining that some of the dilemmas might sound unreal to the participants but that, in spite of this, there were serious philosophical reasons to include them. Participants were asked to ignore this unrealistic nature of some of the dilemmas and to concentrate only on the judgment. A third possibility is to control for the normality of the dilemma situation from the outset; either keeping normality constant across the stimuli or varying it systematically. The latter solution is obviously not applicable in case the goal is to carry out a cross-cultural study, where normality of harm might be a rather relative variable. While cultural differences should be taken into account for all

13 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) the variables mentioned in this review, in this particular case it is especially pressing. A fourth, maybe more holistic, way to address this challenge is to start out by focusing on transgressions that happen to be universally considered as such across different cultures, such as life-and-death transgressions (Hollos et al., 1986; Song et al., 1987). Obviously, the question about the normality of harm relates to a more general discussion about the ecological validity and justification of moral dilemmas: while some authors merely suggest that an effort should generally be made to enhance ecological validity of stimuli in studies on moral cognition without giving any specific suggestions about how this could be achieved (Casebeer, 2003), others express serious doubts about any use of moral dilemma sets in studies on moral judgment, specifically due to their heterogeneity in so many design variables (McGuire et al., 2009). We agree with the latter in that the design variables of moral dilemmas should be carefully controlled, and we believe this can be achieved; in fact, that s precisely the thrust of the present paper. The first remark about ecological validity, however, deserves a second look. Ecological validity, yes, but of what kind? As pointed out, moral judgments to hypothetical moral dilemmas allow us to obtain valuable information about underlying psychological processes (Moore et al., 2011a,b) and we believe that, specifically, far-away situations involving serious moral transgressions are particularly valuable in serving that aim. They allow us to control for nondesired previous differential exposure effects or to specifically control the level of previous exposure to extreme situations, and to investigate what effect this has on moral judgment. By choosing more extreme situations more reliable conclusions can be drawn about the underlying moral intuitions that trigger the distinct moral judgments. We believe this is a valuable strategy for avoiding confounds, a position also acknowledged by other authors (Borg et al., 2006). Thus, dilemmas foster, rather than diminish, ecological validity. To conclude this section, we articulate a methodological proposal which might help to round off this eternal discussion about the legitimacy of moral dilemmas as experimental stimuli. In line with the brief discussion on the use of moral dilemmas in the introductory section of the present paper, we propose to include some indications about the kind of stimuli in the instructions to the participants to make them more easily engage in the task. For instance, by informing them that the dilemmas they are about to read are similar to those that are likely to appear as press releases on TV, in the newspaper, and on the radio, or which might be a plot in a novel or a movie: in the following you will read a series of short stories about difficult interpersonal situations, similar to those that we all see on the news every day or may read about in a novel Certainty of events In real-life scenarios, full certainty about what will happen, and whether or not a moral transgression would or not actually change the course of events, is the exception rather than the rule. Dilemma stories, however, are written in an explicit style that tries to block out any alternative course of action that could occur through unforeseen events and which would spare the participant from making the fateful choice. However, it is possible that the experimental participant still feels that the course of events as depicted is unlikely, in spite of such explicit inevitability formulations. A possible strategy to cope with such uncertainty is to take it into account as a covariate in data analysis as the probability with which the participant believes that the moral transgression would prevent the first depicted harm to happen. Can the body of a person stop a trolley? Why not jumping oneself, then? (As discussed in Subsection 4.3 regarding the kinship variable). Why not try to alert the potential victims to quit the track? Some studies have pointed out the relevance of this variable by asking their participants to rate this probability and to incorporate it in their data analysis (Royzman and Baron, 2002), or again by merely including an allusion to this problem in the instructions, simply asking the participant to ignore it (Greene et al., 2009). We believe, however that the certainty of events is a crucial variable to account for. It is very relevant for making a moral judgment and for the attribution of moral responsibility to know whether there are alternatives to the proposed course of action leading to the moral transgression. In the moment that there is uncertainty of the course of events, the objective of eliciting moral judgments based on the inevitability of a situation is corrupted. If an experimental participant sees a situation as inevitably leading to a moral transgression she might be more inclined to consenting it a position acknowledged also in many legal systems: if committed harm was inevitable, the legal judgment of the action becomes less severe. However, if the participants are confused by the formulation of the dilemma because it leaves room for speculation about possible alternatives, they will be judging a different scenario, no matter how explicitly they are asked to ignore the alternatives. Hence, ideally, either such uncertainty in the formulation should be kept to a minimum, or it should be manipulated as another independent variable. 6. Schematic overview of 25 studies in relation to the 19 design parameters This review provides a review and discussion of 19 experimental design parameters that are relevant in moral dilemma tasks, both for behavioral and neuroscientific approaches to moral psychology. Unfortunately, research to date has not been successful in manipulating them individually while controlling for the rest. Table 1 graphically summarizes which variables have received attention by a selection of 25 different studies appeared in peer-reviewed journals mainly in the past 10 years. In these 25 studies reported in the table (nine of which are reported twice because they belong to two distinct categories on the vertical axis) we observe the following distribution. In the methodology section, there is a striking heterogeneity in terms of the question type in the studies. Between the 25 studies, there are at least 10 different question types. The presentation format (see Subsection 3.1) is mainly computerized (72% of the studies), followed by the pen-and-paper versions (20%) and the web-based modality (8%). Fifty-six percent of the studies are behavioral, 32% are neuroimaging, and 12% lesion studies. As to the participant perspective, 56% of the studies used the first person perspective, 36% the third person perspective and 8% combined the two (see Subsection 3.8 for a discussion of the different perspectives). In relation to the remaining variables we may summarize some of the most important issues as follows. Seven of the 25 studies included justifications as a part of their experimental paradigm 3.9). Less than half of the studies controlled explicitly for the variable of intentionality (40%), in spite of the importance of this variable in moral judgment formation (see Subsection 5.1). However, most of the other studies have simply avoided any reference to intentions in their dilemma formulation. Conversely, the directness of harm variable was the one most systematically controlled for 5.3). Ninety-two percent of the studies take this variable into account. Also the trade-off has received much attention. Sixty-four percent of the studies control explicitly for this variable but 28% of the studies fail to control for it, possibly biasing their results (see Subsection 5.4). The type of transgression varies unsystematically in 40% of the studies, whereas 60% control for it. Forty percent is a lot, considering that this means that the authors have been comparing judgments of social norm

14 1262 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) transgressions such as lying, stealing etc. with serious bodily harm such as rape, injure or killing (see Subsection 5.2 for a discussion on the difference between the types of harm). The Kin-/friendship variable varies unsystematically in 56% of the studies, which is worrying considering that the 16% of studies who systematically control for this variable report consistently how important this variable is for moral decision-making as individuals appear to favor rescuing themselves, their families and friends if they have the choice between them and strangers (see Subsection 4.3). A similarly preoccupying pattern emerges for the related variable of in-/out group: in 60% of the studies it varies unsystematically and only 16% control for it 4.2). Furthermore, in spite of their relevance (see Subsections 5.5 and 5.6), the variables of certainty of events and the normality of situations are hardly controlled for. It is also striking, how few studies have controlled for the word number count (60% did not) or the expression style (60% did not) (see the Subsections 3.4 and 3.2 respectively). 7. Conclusion and discussion 7.1. Conclusion Our main conclusion derived from this review is that using moral dilemmas in Neuroethics has much to contribute to our understanding of human moral psychology. We advocate that a moral dilemma should be understood as an experimental stimulus with a clear structure composed of a series of design parameters independent variables that have to be identified and then carefully controlled for in all empirical studies of moral judgment using this methodology. We have specifically not aimed to make a review of the neuroscientific findings of studies on moral cognition. For this, we refer the interested reader to a series of very thoroughly crafted reviews already available in the literature (Casebeer, 2003; Dean, 2009; Greene and Haidt, 2002; Hauser and Young, 2008; Moll et al., 2003; Moll and Schulkin, 2009; Woodward and Allman, 2007; Young and Koenigs, 2007). Rather, we have offered a principled account of the issues involved in using this methodology, identifying a number of questions which have barely received experimental attention, as well as others for which somewhat firmer empirical grounds are available. For progress at the theoretical level, the currently most articulated proposals in this area the Dual Process Hypothesis of Moral Judgment (DPHMJ) (Greene et al., 2001), the Motivational Approach (Moll et al., 2008a,b), and the Five Foundations Account (Haidt and Graham, 2007) can be developed in this regard, to see how they fare when this framework of parameters is taken into account, and whether they make different predictions on particular issues that can then be put to the test Future directions: a theoretical framework for working hypotheses The Dual Process Hypothesis of Moral Judgment (DPHMJ) is an example of a model of human morality that attempts to provide an account of what it is that makes us diverge in our moral judgments (Greene, 2008; Greene et al., 2001). It posits that both top down reasoning processes of a more cognitive nature and bottom up emotionally triggered processes interact in moral judgment formation. However, this appears as a rather too dichotomic view of the processes involved in view of the evidence reported throughout this review, considering the wealth of contextual and conceptual variables whose importance in moral judgment formation is undeniable. The DPHMJ model does not explicitly consider these variables. It thus ignores important psychological and social processes. Nevertheless, the DPHMJ has provided a valuable starting point for building an integrative model of human moral judgment. In 2008, Moll and colleagues proposed a Motivational Approach on how moral intuitions, moral emotions and values might be interacting to motivate the unique phenomenon of morality in human cognition. One of their main claims is that our social behavior (or morality) is determined by biological predispositions (intuitions) that trigger moral emotions which are of key importance for modulating subsequent behaviors. These predispositions evolved during human evolutionary history as motivational forces to foster prosocial behavior, paralleling the demands of a life in increasingly large social groups (Moll et al., 2008a,b). While these authors do not specify in detail what these biological predispositions might be, considering the work by Haidt and colleagues may complement the picture. The Foundations approach of human morality (Haidt and Graham, 2007) suggests five such basic predispositions: harm/care, fairness/justice, ingroup loyalty, authority/respect, purity/sanctity. According to this approach, through a complex interplay of social, cognitive and emotional processes, each of these foundations condition human moral behavior, and, a judgment that an individual makes in psychological terms is also a type of behavior. Accordingly, these three models have the potential to serve as a theoretical framework for future studies on moral judgment with moral dilemmas. In the foundations view, transgressions towards any of the five intuitions trigger moral emotions in the individual, and, according to the motivational view, these will motivate one type of behavior (here: judgment) or another. The judgment will depend on a variety of parameters of the situation as discussed in this review which again are thoroughly covered by the foundations view. In consequence, we believe that the Five Foundations Model of morality and the Motivational Approach, interwoven with available evidence on emotions and moral emotions (for instance, provided by the DPHMJ), and on how they interact to condition behavior, constitute a valuable theoretical framework for future studies on moral judgment. Based on the evidence derived from studies with moral dilemmas that vary in methodological and conceptual design parameters discussed in this review, we suggest that moral dilemmas, specifically, are a highly valuable tool for assessing human moral cognition. It is the possibility of varying so systematically the variables implied in a situation that makes moral dilemmas such a profitable experimental paradigm in empirical and neuroscientific studies of moral psychology. We hope that this review and the suggestions based on it will make it possible to fully exploit that value in future research. Acknowledgements The study was funded by the research project FFI (Spanish Ministry of Science and Innovation), and by the Chair of the Three Religions (Government of the Balearic Islands) of the University of the Balearic Islands, Spain. Julia Frimodt Christensen was supported by a FPU PhD scholarship from the Spanish Ministry of Education (AP ). A special thank you goes to Dr Marcus Pearce and Dr Marcos Nadal for very helpful comments on previous drafts of the manuscript. We would also like to thank the referees for their very valuable comments. References Bartels, D.M., Principled moral sentiment and the flexibility of moral judgment and decision making. Cognition 108 (2), , doi: /j.cognition Berthoz, S., Armony, J.L., Blair, R.J.R., Dolan, R.J., An fmri study of intentional and unintentional (embarrassing) violations of social norms. Brain 125,

15 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) Berthoz, S., Grezes, J., Armony, J.L., Passingham, R.E., Dolan, R.J., Affective response to one s own moral violations. NeuroImage 31 (2), , doi: /j.neuroimage Bjorklund, F., Differences in the justification of choices in moral dilemmas: effects of gender, time pressure and dilemma seriousness. Scandinavian Journal of Psychology 44 (5), Borg, J.S., Hynes, C., Van Horn, J., Grafton, S., Sinnott-Armstrong, W., Consequences, action, and intention as factors in moral judgments: an fmri investigation. Journal of Cognitive Neuroscience 18 (5), Burnstein, E., Crandall, C., Kitayama, S., Some neo-darwinian decision rules for altruism weighing cues for inclusive fitness as a function of the biological importance of the decision. Journal of Personality and Social Psychology 67 (5), Cacioppo, J.T., Petty, R.E., Kao, C.F., The efficient assessment of need for cognition. Journal of Personality Assessment 48 (3), , doi: /s jpa Casebeer, W.D., Moral cognition and its neural constituents. Nature Reviews Neuroscience 4 (10), , doi: /nrn1223. Ciaramidaro, A., Adenzato, M., Enrici, I., Erk, S., Pia, L., Bara, B.G., et al., The intentional network: how the brain reads varieties of intentions. Neuropsychologia 45 (13), , doi: /j.neuropsychologia Cikara, M., Farnsworth, R.M., Harris, L.T., Fiske, S.T., On the wrong side of the trolley track: neural correlates of relative social valuation. Social Cognitive and Affective Neuroscience 5 (4), , doi: /scan/nsq011. Crowne, D.P., Marlowe, D., A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology 24 (4), , doi: /h Cushman, F., Crime and punishment: distinguishing the roles of causal and intentional analyses in moral judgment. Cognition 108 (2), , doi: /j.cognition Cushman, F., Young, L., Hauser, M., The role of conscious reasoning and intuition in moral judgment: testing three principles of harm. Psychological Science 17 (12), Cvencek, D., Greenwald, A.G., Brown, A.S., Gray, N.S., Snowden, R.J., Faking of the implicit association test is statistically detectable and partly correctable. Basic and Applied Social Psychology 32 (4), , doi: / Damasio, A.R., Consciousness knowing how, knowing where. Nature 375 (6527), Dean, R., Does neuroscience undermine deontological theory? Neuroethics 3 (1), 43 60, doi: /s x. Edwards, A.L., Horst, P., Social desirability as a variable in l technique studies. Educational and Psychological Measurement 13 (4), , doi: / Farrer, C., Frith, C.D., Experiencing oneself vs another person as being the cause of an action: the neural correlates of the experience of agency [Article]. NeuroImage 15 (3), , doi: /nimg Ferstl, E.C., Neumann, J., Bogler, C., von Cramon, D.Y., The extended language network: a meta-analysis of neuroimaging studies on text comprehension. Human Brain Mapping 29 (5), , doi: /hbm Finger, E.C., Marsh, A.A., Kamel, N., Mitchell, D.G.V., Blair, J.R., Caught in the act: the impact of audience on the neural response to morally and socially inappropriate behavior. NeuroImage 33 (1), , doi: /j.neuroimage Fiske, S.T., Cuddy, A.J.C., Glick, P., Universal dimensions of social cognition: warmth and competence. Trends in Cognitive Sciences 11 (2), 77 83, doi: /j.tics Fiske, S.T., Cuddy, A.J.C., Glick, P., Xu, J., A model of (often mixed) stereotype content: competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology 82 (6), , doi: // Foot, P., The problem of abortion and the doctrine of the double effect. In: Reprinted in Virtues and Vices and Other Essays in Moral Philosophy. Blackwell, Oxford, pp Franklin, S.A., McNally, R.J., Riemann, B.C., Moral reasoning in obsessivecompulsive disorder [Article]. Journal of Anxiety Disorders 23 (5), , doi: /j.janxdis Fumagalli, M., Ferrucci, R., Mameli, F., Marceglia, S., Mrakic-Sposta, S., Zago, S., et al., Gender-related differences in moral jugdments. Cognitive Processing 11 (3), , doi: /s Fumagalli, M., Vergari, M., Pasqualetti, P., Marceglia, S., Mameli, F., Ferrucci, R., et al., Brain switches utilitarian behavior: does gender make the difference? Plos One 5 (1), doi: /journal.pone Gomila, A., Suicide terrorists, neuroscience, and morality: taking complexities into account. In: Vilarroya, O., Forn i Argimon, F. (Eds.), Social Brain Matters., 1st ed. Radopi, Amsterdam. Gomila, A., Calvo, P., Understanding brain circuits and their dynamics. Behavioral and Brain Sciences 33, Graham, J., Haidt, J., Nosek, B.A., Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology 96 (5), , doi: /a Greene, J., The secret Joke of Kant s Soul. In: Sinnott-Armstrong, W. (Ed.), Moral Psychology, Vol. 3. MIT Press, Cambridge, Massachusetts; London, England, pp Greene, J.D., Dual-process morality and the personal/impersonal distinction: a reply to McGuire, Langdon, Coltheart, and Mackenzie. Journal of Experimental Social Psychology 45 (3), , doi: /j.jesp Greene, J., Haidt, J., How (and where) does moral judgment work? Trends in Cognitive Sciences 6 (12), Greene, J., Sommerville, B., Nystrom, L., Darley, J., Cohen, J., Cognitive and affective conflict in moral judgment. Journal of Cognitive Neuroscience, 49. Greene, J.D., Cushman, F.A., Stewart, L.E., Lowenberg, K., Nystrom, L.E., Cohen, J.D., Pushing moral buttons: the interaction between personal force and intention in moral judgment. Cognition 111 (3), , doi: /j.cognition Greene, J.D., Morelli, S.A., Lowenberg, K., Nvstrom, L.E., Cohen, J.D., Cognitive load selectively interferes with utilitarian moral judgment. Cognition 107 (3), , doi: /j.cognition Greene, J.D., Nystrom, L.E., Engell, A.D., Darley, J.M., Cohen, J.D., The neural bases of cognitive conflict and control in moral judgment. Neuron 44 (2), Greene, J.D., Sommerville, R.B., Nystrom, L.E., Darley, J.M., Cohen, J.D., An fmri investigation of emotional engagement in moral judgment. Science 293 (5537), Greenwald, A.G., McGhee, D.E., Schwartz, J.L.K., Measuring individual differences in implicit cognition: the implicit association test. Journal of Personality and Social Psychology 74 (6), , doi: / Greenwald, A.G., Nosek, B., Banaji, M.R., Understanding and using the implicit association test: 1. An improved scoring algorithm. Journal of Personality and Social Psychology 85, Grice, H.P., Logic and conversation. In: Cole, P., Morgan, J.L. (Eds.), Syntax and Semantics 3: Speech Acts. Academic Press, New York. Haidt, J., The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychological Review 108 (4), , doi: // x Haidt, J., Elevation and the positive psychology of morality. In: Keyes, C.L., Haidt, J. (Eds.), Flourishing: Positive Psychology and the Life Well-Lived. American Psychological Association, Washington, DC, pp Haidt, J., Graham, J., When morality opposes justice: conservatives have moral intuitions that liberals may not recognize. Social Justice Research 20 (1), Hashimoto, R., Sakai, K.L., Specialization in the left prefrontal cortex for sentence comprehension. Neuron 35 (3), , doi: /s (02) Hauser, M., Cushman, F., Young, L., Jin, R.K.X., Mikhail, J., A dissociation between moral judgments and justications. Mind & Language 22 (1), Hauser, M.D., Young, L., Modules, minds and morality. In: Pfaff, D., Kordon, C., Chanson, P., Christen, Y. (Eds.), Hormones and Social Behavior., pp Heekeren, H.R., Wartenburger, I., Schmidt, H., Schwintowski, H.P., Villringer, A., An fmri study of simple ethical decision-making. Neuroreport 14 (9), , doi: /01.wnr a7. Hewstone, M., Rubin, M., Willis, H., Intergroup bias. Annual Review of Psychology 53, Hofmann, W., Baumert, A., Immediate affect as a basis for intuitive moral judgement: an adaptation of the affect misattribution procedure. Cognition & Emotion 24 (3), , doi: / Hollos, M., Leis, P., Turiel, E., Social reasoning in Ijo children and adolescents in Nigerian communities. Journal of Cross-Cultural Psychology 17 (3), , doi: / Karniol, R., Children s use of intention cues in evaluation behavior. Psychological Bulletin 85, Keasey, C.B., Children s developing awareness and usage of intentionality and motive. In: Keasey, C.B. (Ed.), Nebraska Symposium on Motivation, Vol. 25. University of Nebraska Press, Lincoln, pp Killgore, W.D.S., Killgore, D.B., Day, L.M., Li, C., Kamimori, G.H., Balkin, T.J., The effects of 53 hours of sleep deprivation on moral judgment. Sleep 30 (3), Koenigs, M., Young, L., Adolphs, R., Tranel, D., Cushman, F., Hauser, M., et al., Damage to the prefrontal cortex increases utilitarian moral judgements. Nature 446 (7138), , doi: /nature Kohlberg, L., Development of moral character and moral ideology. In: Hoffman, M.L., Hoffman, L.W. (Eds.), Review of Child Development Research, Vol. 1. Russell Sage Foundation, New York. Kraut, R., Olson, J., Banaji, M.R., Cohen, J., Cooper, M., Psychological research online. American Psychologist 59 (2), , doi: / x Lanteri, A., Chelini, C., Rizzello, S., An experimental investigation of emotions and reasoning in the trolley problem. Journal of Business Ethics 83 (4), , doi: /s Lombrozo, T., The role of moral commitments in moral judgment. Cognitive Science 33 (2), , doi: /j x. Love, T., Haist, F., Nicol, J., Swinney, D., A functional neuroimaging investigation of the roles of structural complexity and task-demand during auditory sentence processing. Cortex 42 (4), , doi: /s (08) McGuire, J., Langdon, R., Coltheart, M., Mackenzie, C., A reanalysis of the personal/impersonal distinction in moral psychology research. Journal of Experimental Social Psychology 45 (3), , doi: /j.jesp Mikhail, J., Universal moral grammar: theory, evidence and the future. Trends in Cognitive Sciences 11 (4), , doi: /j.tics

16 1264 J.F. Christensen, A. Gomila / Neuroscience and Biobehavioral Reviews 36 (2012) Miller, J.G., Bersoff, D.M., The role of liking in perceptions of the moral responsibility to help: a cultural perspective. Journal of Experimental Social Psychology 34 (5), Moll, J., de Oliveira-Souza, R., Bramati, I.E., Grafman, J., Functional networks in emotional moral and nonmoral social judgments. NeuroImage 16 (3), , doi: /nimg Moll, J., de Oliveira-Souza, R., Eslinger, P.J., Morals and the human brain: a working model. Neuroreport 14 (3), , doi: /01.wnr Moll, J., de Oliveira-Souza, R., Moll, F.T., Ignacio, F.A., Bramati, I.E., Caparelli-Daquer, E.M., et al., The moral affiliations of disgust a functional MRI study. Cognitive and Behavioral Neurology 18 (1), Moll, J., de Oliveira-Souza, R., Zahn, R., 2008a. The neural basis of moral cognition sentiments, concepts, and values. Year in Cognitive Neuroscience , , doi: /annals Moll, J., Oliveira-Souza, R., Zahn, R., Grafman, J., 2008b. The cognitive neuroscience of moral emotions. In: Sinnott-Armstrong (Ed.), Moral Psychology, Vol. 3. MIT Press, Cambridge, Massachusetts; London, England, pp Moll, J., Schulkin, J., Social attachment and aversion in human moral cognition. Neuroscience and Biobehavioral Reviews 33 (3), , doi: /j.neubiorev Moore, A.B., Clark, B.A., Kane, M.J., Who shalt not kill? Individual differences in working memory capacity, executive control, and moral judgment. Psychological Science 19 (6), Moore, A.B., Lee, N.Y.L., Clark, B.A.M., Conway, A.R.A., 2011a. In defence of the personal/impersonal distinction in moral psychology research: cross-cultural validation of the dual process model of moral judgment. Judgment and Decision Making 6 (3), Moore, A.B., Stevens, J., Conway, A.R.A., 2011b. Individual differences in sensitivity to reward and punishment predict moral judgment. Personality and Individual Differences 50 (5), , doi: /j.paid Nelson, T.D. (Ed.), The Psychology of Prejudice., 2nd ed. Allyn & Bacon, Bosten. Nichols, S., Knobe, J., Moral responsibility and determinism: the cognitive science of folk intuitions. Nous 41 (4), , doi: /j x. Nichols, S., Mallon, R., Moral dilemmas and moral rules. Cognition 100 (3), , doi: /j.cognition Nosek, B., Banaji, M.R., Greenwarld, A.G., Harvesting intergroup attitudes and stereotypes from a demostration website. Group Dynamics 6, O Hara, R.E., Sinnott-Armstrong, W., Sinnott-Armstrong, N.A., Wording effects in moral judgments. Judgment and Decision Making 5 (7), O Neill, P., Petrinovich, L., A preliminary cross-cultural study of moral intuitions. Evolution and Human Behavior 19 (6), Peelle, J.E., McMillan, C., Moore, P., Grossman, M., Wingfield, A., Dissociable patterns of brain activity during comprehension of rapid and syntactically complex speech: evidence from fmri. Brain and Language 91 (3), , doi: /j.bandl Petrinovich, L., O Neill, P., Influence of wording and framing effects on moral intuitions. Ethology and Sociobiology 17 (3), Petrinovich, L., O Neill, P., Jorgensen, M., An empirical-study of moral intuitions toward an evolutionary ethics. Journal of Personality and Social Psychology 64 (3), Phillips, S.T., Ziller, R.C., Toward a theory and measure of the nature of nonprejudice. Journal of Personality and Social Psychology 72, Piaget, J. (Ed.), 1932/1965. The Moral Judgment of the Child. Free Press, New York. Pizarro, D.A., Uhlmann, E., Bloom, P., Causal deviance and the attribution of moral responsibility. Journal of Experimental Social Psychology 39 (6), , doi: /s (03) Quinn, W., Actions, intentions and the doctrine of doing and allowing. Philosophical Review 98, Rabbie, J.M., The effects of intergroup competition and cooperation on intraand intergroup relationships. In: Grzelak, J., Derlega, V. (Eds.), Living with Other People: Theory and Research on Cooperation and Helping. Academic Press, New York. Richman, W.L., Kiesler, S., Weisband, S., Drasgow, F., A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology 84 (5), , doi: / Ritov, I.B.J., Reluctance to vaccinate: omission bias and ambiguity. Journal of Behavioral Decision Making 3, Royzman, E., Baron, J., The preference of indirect harm. Social Justice Research 15 (2), Sabin, J.A., Nosek, B.A., Greenwald, A.G., Rivara, F.P., Physicians implicit and explicit attitudes about race by MD race, ethnicity, and gender. Journal of Health Care for the Poor and Underserved 20 (3), Saxe, R., Powell, L.J., It s the thought that counts: specific brain regions for one component of theory of mind. Psychological Science 17 (8), , doi: /j x. Schnabel, K., Asendorpf, J.B., Greenwald, A.G., 2008a. Assessment of individual differences in implicit cognition. A review of IAT measures. European Journal of Psychological Assessment 24 (4), , doi: / Schnabel, K., Asendorpf, J.B., Greenwald, A.G., 2008b. Using implicit association tests for the assessment of implicit personality self-concepts. In: Boyle, G.j., Matthews, G., Saklofske, D.H. (Eds.), Handbook of Personality Theory and Testing. Sage, London, pp Shaver, K.G. (Ed.), The Attribution of Blame: Causality, Responsibility, and Blameworthiness. Springer-Verlag, New York. Shweder, R.M.N., Mahapatra, M., Park, L., The Big Three of morality (autonomy, community, divinity) and the Big Three explanations of suffering. In: Rozin, A.M.B.P. (Ed.), Morality and Health. Routledge, Oxford, pp Song, M., Smetana, J., Kim, S., Korean children s conceptions of moral and conventional transgressions. Developmental Psychology 23, Sorensen, R.A., Blindspots. Oxford University Press, Oxford. Styron, T. (Ed.), Sophie s Choice. Random House, New York. Takahashi, H., Yahata, N., Matsuda, T., Asai, K., Okubo, Y., Brain activation associated with evaluative processes of guilt and embarassment: an fmri study. NeuroImage 23, Takezawa, M., Gummerum, M., Keller, M., A stage for the rational tail of the emotional dog: roles of moral reasoning in group decision making. Journal of Economic Psychology 27 (1), , doi: /j.joep Tangney, J.P., Stuewig, J., Mashek, D.J., Moral emotions and moral behavior. Annual Review of Psychology 58, , doi: /annurev.psych Thaler, R., Mental accounting and consumer choice. Marketing Science 27, Thomson, J.J., Killing, letting die, and the trolley problem. The Monist 59, Turiel, E. (Ed.), The Development of Social Knowledge: Morality & Convention. Cambridge University Press, New York. Turiel, E., Killen, M., Helwig, C., Morality: its structure, functions, and vagaries. In: Kagan, J., Lamb, S. (Eds.), The Emergence of Morality in Young Children. University of Chicago Press, Chicago, pp Tversky, A., Kahneman, D., The framing of decisions and the psychology of choice. Science 211 (4481), , doi: /science Valdesolo, P., DeSteno, D., Manipulations of emotional context shape moral judgment. Psychological Science 17 (6), Waldmann, M.R., Dieterich, J.H., Throwing a bomb on a person versus throwing a person on a bomb intervention myopia in moral intuitions. Psychological Science 18 (3), Wang, X.T., Evolutionary hypotheses of risk-sensitive choice: age differences and perspective change. Ethology and Sociobiology 17 (1), Weiner, B., Peter, N., A cognitive-developmental analysis of achievement and moral judgments. Developmental Psychology 9, Williams, B., A critique of utilitarianism. In: Smart, J.J.C., Williams, B. (Eds.), Utilitarianism: For and Against. Cambridge University Press, Cambridge. Woodward, J., Allman, J., Moral intuition: its neural substrates and normative significance. Journal of Physiology-Paris 101 (4 6), , doi: /j.jphysparis Young, L., Koenigs, M., Investigating emotion in moral cognition: a review of evidence from functional neuroimaging and neuropsychology. British Medical Bulletin 84, 69 79, doi: /bmb/ldm031. Young, L., Saxe, R., The neural basis of belief encoding and integration in moral judgment. NeuroImage 40 (4), , doi: /j.neuroimage Young, L., Saxe, R., 2009a. An fmri investigation of spontaneous mental state inference for moral judgment. Journal of Cognitive Neuroscience 21 (7), Young, L., Saxe, R., 2009b. Innocent intentions: a correlation between forgiveness for accidental harm and neural activity. Neuropsychologia 47 (10), , doi: /j.neuropsychologia Zahn, R., Moll, J., Paiva, M., Garrido, G., Krueger, F., Huey, E.D., et al., The neural basis of human social values: evidence from functional MRI. Cerebral Cortex 19 (2), , doi: /cercor/bhn080. Zelazo, P.D., Helwig, Ch.C., Lau, A., Intention, act, and outcome in behavioral prediction and moral judgment. Child Development 67,