Changes in behavior-related neuronal activity in the striatum during learning

Transcription

1 Review Vol.26 No.6 June Changes in behavior-related neuronal activity in the striatum during learning Wolfram Schultz, Léon Tremblay and Jeffrey R. Hollerman Institute of Physiology, University of Fribourg, CH-1700 Fribourg, Switzerland, and Department of Anatomy, University of Cambridge, Cambridge CB2 3DY, UK The involvement of the striatum in numerous forms of learning and memory is likely to be based on changes in neuronal activity when specific behavioral tasks are being learned. Striatal neurons show distinctive changes when animals learn the significance of stimuli that predict rewards and induce the preparation of movements. These changes resemble some of the simultaneous, learning-related changes in closely associated areas of the frontal cortex. The striatal changes might assist in adapting existing reward expectations and behaviors to novel or changing environmental conditions and they could contribute to the functions of the basal ganglia in learning, reward expectation and movement preparation. The mammalian striatum (i.e. the caudate nucleus, putamen and ventral striatum, including nucleus accumbens) is an important brain structure involved in controlling behavioral output. It is closely associated, through multiple convergent and partly closed loops, with all areas of the cerebral cortex. Despite 30 years of intense research and numerous suggestions, still no unifying concept of striatal function has been accepted. The lateral putamen is activated during movements, and parkinsonian patients with dopamine deficiencies in the putamen have severe deficits in motor activity. The rostral caudate nucleus and putamen have prominent connections with most areas of the association cortex ( associational striatum ). Many parts of the striatum, including the ventral striatum (nucleus accumbens), are involved in reward processing. Synaptic connections in the striatum show use-dependent plasticity [1], suggesting involvement in various forms of learning and memory, such as procedural learning, sensorimotor and skill learning [2,3], habit learning [4,5], goal-directed-instrumental and reward-association learning [6], and emotional learning [7]. Without attempting to resolve all mysteries of striatal function, this review describes (referring primarily to the monkey striatum) how neuronal activity changes in the striatum during learning of behavioral tasks and how these changes compare with those in the frontal cortex. as a change in predictions Many learning forms involve setting up predictions, and the progress of learning depends on reducing the Corresponding author: Wolfram Schultz (ws234@cam.ac.uk). difference between the predicted and actual occurrence of events. Using predictions for anticipating the future appears to be how many parts of the brain work. Whereas sensory systems use information received from the outside world through specific organs, brain structures involved in organizing behavioral outputs need to predict to some extent what will happen in the future. Incorporated in these predictions is often a representation of how the action of the organism will influence the world to reach the goal of behavior. The organism thus economizes by not processing all incoming information, instead making predictions and focusing processing on deviations from the predictions. is manifested as a reproducible change of behavior in identical situations. It can be viewed as a change in predictions of outcomes (e.g. reward, punishment, behavioral reactions, external stimuli or internal states). Outcomes that are different from the prediction modify behavior in a direction that reduces the discrepancy between the outcome and its prediction until the outcome and prediction match. This concept applies to classical (Pavlovian) conditioning, in which learning consists of changing the prediction until it matches the outcome and behavioral changes are consequential to the prediction change [8,9]. In instrumental (operant) conditioning, the behavioral action is performed with some prediction of outcome. consists in changing the behavior until the outcome matches the prediction. Thus, setting up and modifying predictions are essential for basic neuronal functioning and for adapting behavior to the requirements of the organism. Responses to stimuli Discriminative visual and auditory stimuli inform animals on how and when to react and which rewards to expect. The behavioral and motivational significances of the stimuli are learned over several weeks and months and are stored in long-term memory. Neurons in the striatum and frontal cortex show phasic responses to these stimuli (Fig. 1a), thus discriminating spatial movement targets [10], sensory object properties [11,12] and types of reward [13,14]. Acquisition of responses The typical movement-instructing and reward-predicting responses in the striatum are acquired through repeated experience of the reward and the movement necessary for /03/$ - see front matter q 2003 Elsevier Science Ltd. All rights reserved. doi: /s (03)00122-x

2 322 Review Vol.26 No.6 June 2003 (a) Response to event 1 (b) Memory of event 1 (c) Expectation of event 2 Event 1 Event 2 Time Fig. 1. Schematics of neuronal activation. (a) Response to learned stimuli (event 1) with behavioral or motivational significance. (b) Maintenance of working memory. (c) Expectation or preparation of a predictable event (event 2). obtaining it. Through learning, neurons in the rat sensorimotor striatum become activated at the start and finish of T-maze performance through learning [15]. The depressant, discriminatory responses of tonically active striatal interneurons (TANs) to reward-predicting auditory stimuli are rare before learning but they become substantial following 15 min of repeated pairing with liquid [16]. Behavioral extinction greatly reduces the response within 10 min. Acquisition and expression of this neuronal response to reward-predicting stimuli are abolished by interference with dopamine neurotransmission [17]. Responses are reinstated by systemic administration of the dopamine agonist apomorphine. During repeated learning (Box 1), the slowly firing striatal neurons develop activating responses to movementinstructing and reward-predicting stimuli during individual learning episodes [18]. Thus, the acquisition of discriminatory responses in the striatum reflects the learning of appropriate behavioral responses. Transient changes during learning Single neurons can be studied over entire learning episodes in learning-set paradigms (Box 1) [18 21]. Approximately 20% of task-related, slowly firing neurons in the rostral striatum show transient responses during the learning of novel instruction or movement-triggering stimuli (Fig. 2) [18]. Neurons in the tail of the caudate nucleus show stronger visual responses to novel stimuli than to well learned ones [22]. Responses can occur also in additional trial types during initial learning, suggesting a Box 1. A controlled learning paradigm Rather than studying the learning of a new task from scratch, it is possible to train animals to learn single task components at a time. The learning of hundreds of novel stimuli results in a learning set in which learning occurs rapidly. Activity from the same striatal neuron is studied during an entire learning episode and compared with performance using familiar instructions. The animal is first trained in a behavioral task that tests some of the functions of the brain structure under study (Fig. Ia). Striatal neurons show differential activations during individual components of the task (Fig. Ib). Most neuronal activations depend on reward and are absent in predictably unrewarded trials. When novel instruction pictures are introduced, learning occurs largely within the first 5 10 trials (Fig. Ic). In familiar trials, the initial instruction allows the animal to prepare the behavioral reaction and predicts the reward. With stimuli that are novel and based on the pre-learned task context, animals show behavioral signs of reward expectation in all initial movement trials and subsequently adapt the expectations according to rewarded versus unrewarded movements. This procedure resembles everyday situations in which only a minimum of new information is learned and most other expectations remain valid or are only slightly altered. (a) Movement Reward No movement Reward Movement No reward (b) Movement instruction Movement preparation Movement initiation s Instruction Trigger React Reward expectation Fig. I. task and basic neuronal activity. (a) Delayed response task. An initial picture presented on a computer monitor (at time 0 s) instructs the animal to perform a rewarded movement, unrewarded movement or rewarded non-movement reaction following a subsequent uniform trigger stimulus (e.g. being shown a red square). Small drops of juice (reward, indicated by black drop-shapes) or a sound serve as reinforcing signals if the trial is correctly performed. For learning, three novel instruction pictures are used for the three trial types, and the animal finds out by trial and error which of the novel pictures corresponds to the known trial types. (b) The main forms of behavioral relationships of striatal neurons, green indicating when striatal neurons are active. (c) curves indicating rapid learning of the three trial types shown in (a). Part (c) reproduced, with permission, from Ref. [18], q (1998) The American Physiological Society. Reward detection Instruction Movement Reward (c) Correct performance (%) Rewarded movement Rewarded non-movement Unrewarded movement trials

3 Review Vol.26 No.6 June (a) (b) s s Instruction Trigger Trigger Reward Fig. 2. Transient responses in two caudate neurons (a,b) during learning. (a) Long-lasting response to the instruction during learning, with gradual disappearance during subsequent trials. (b) Short-lasting response to the movement-triggering stimulus. Dots in rasters denote the time of neuronal impulses. Each line of dots represents one trial. The sequence of trials is plotted chronologically from top to bottom, learning rasters beginning with the first presentations of new instructions. Reproduced, with permission, from Ref. [18], q (1998) The American Physiological Society. transient loss of selectivity during learning. The transient activations might reflect the storage of the specific significances of the stimuli in the memory. Neurons in several areas of the frontal cortex acquire discriminatory responses to movement-instructing and reward-predicting visual, auditory and olfactory stimuli during learning of novel direct or delayed response tasks, with the proportions of responding neurons being comparable with those in the striatum [23 25]. The responses engage progressively greater proportions of neurons during initial learning, consolidation and established performance [23]. In delay tasks comprising repeated learning episodes, frontal cortical neurons fail to respond to novel stimuli but acquire discriminatory responses to a similar extent to that seen in the striatum [19 21,26,27]. In some cases, neuronal responses initially occur indiscriminately to all novel stimuli but differentiate as learning advances [21,28]. During behavioral reversal, frontal cortical neurons show rapid loss of responses to previously learned stimuli and rapid acquisition of specific responses to the alternate stimuli, closely corresponding with behavioral changes [25,26,29]. Some of the neuronal changes during learning and reversal precede slightly the changes in behavior [25,28] and could, thus, be involved in modifying the neural processing that underlies the changed behavior. The transient neuronal activations during learning occur also in frontal cortical neurons ( learning-selective changes ) [18,20,27]. Compared with the striatum, they are twice as frequent in the orbitofrontal cortex [21] but much less frequent in the supplementary and frontal eye fields [20,27], suggesting higher plasticity in orbitofrontal cortex than in more movement-related cortical areas. Changes last usually for more learning trials in the orbitofrontal cortex and supplementary eye field, if learning speed and tasks are comparable. Anticipatory activity The presentation of a learned stimulus can lead to recall of its behavioral and motivational significance from longterm memory. The stimulus can be stored in working memory ( remember what just happened ) and can induce short-term preparations of behavioral actions ( remember what to do ). For example, the behavioral significance of a movement-instructing stimulus is stored in long-term memory and its present occurrence is kept in working memory while the movement is being prepared and executed. Likewise, a reward-predicting stimulus can induce an expectation of the future reward. Expectation and preparation have common anticipatory components, as they precede rather than follow predictable events, but they refer to different predictable events (stimuli and rewards versus action) and, thus, occur on different occasions in discriminating behavioral tasks. The prospective (forward in time) expectation and preparation processes differ schematically from retrospective (backward in time) working memory processes, which concern the previous event and do not require predictive coding (although they are somewhat entangled and could be considered common constituents of working memory) [30]. Neurons in the striatum and

4 324 Review Vol.26 No.6 June 2003 Movement Non-movement s s Instruction Trigger Reward Instruction Trigger Reward Fig. 3. Acquisition of differential movement-preparatory activity during learning. This caudate neuron showed also substantial movement-preparatory activity in familiar movement trials (top). and learning trials (bottom) were run in separate trial blocks but movement and non-movement trials alternated semi-randomly and have been separated for analysis. Trials are plotted chronologically from top to bottom in each raster. Reproduced, with permission, from Ref. [18], q (1998) The American Physiological Society. frontal cortex are activated differentially and specifically during the retrospective encoding, maintenance and recall of working memory [10 12] (Fig. 1b) and during the prospective preparation of movement and expectation of reward [13,14,31 37] (Fig. 1c). Acquisition of anticipatory activity Anticipatory activity develops in some striatal neurons during initial learning trials, until the respective predictions of the novel stimuli have been sorted out by the animal [18] (Fig. 3). This learning change occurs with activities related to the preparation of movement or expectation of reward. The neurons apparently require a learned stimulus to evoke the anticipatory activity. Adaptation of anticipatory activity In learning-set paradigms, the existing task structure is maintained and only the novel stimuli are learned (Box 1). This constancy allows the animal to maintain some basic expectations about the different task events and simply map them onto the novel stimuli. Behavioral reactions to novel stimuli reveal that animals expect reward by default and differentially attribute the existing expectations to the novel stimuli [18]. Rewarded movement Unrewarded movement s s Unrewarded movements Instruction Movement Reward Instruction Movement Sound Return to key Return to key Fig. 4. Adaptation of existing reward expectation during learning. During familiar performance (top), this caudate neuron showed a sustained response to the initial instruction cue in rewarded movement trials and a transient response in non-movement trials (not shown). During learning (bottom), the sustained response occurred initially also in unrewarded movement trials, which the animal performed as if it expected reward (the reward expectation is evidenced when the animal returned its hand to the resting key earlier in rewarded movements than in unrewarded movement trials ( return to key ). The sustained response disappeared when movement parameters became typical for unrewarded movements (arrows to the right). Trials are plotted chronologically from top to bottom in each raster. Reproduced, with permission, from Ref. [18], q (1998) The American Physiological Society.

5 Review Vol.26 No.6 June The adaptation of existing anticipatory neuronal activity occurs together with the behavioral adaptations [18]. Striatal neurons show reward-expectation-related activations during all initial movement trials and these become restricted to rewarded as opposed to unrewarded movements concurrent with, or slightly earlier than, the shift in behavioral indices of expectation (Fig. 4). Similar shifts are seen with the movement preparatory activity that occurs whenever a movement is made and that differentiates once the instructions have been learned (Fig. 5). Striatal neurons show preparatory activity also in erroneous non-movement trials, indicating that the activity reflects the future behavioral reaction and not the (yet unknown) instruction. Thus, the anticipatory activities reflect the expected reward (even if it is incorrectly expected) or the actually prepared movement (even if it goes wrong). These activities appear to be specified in content by the task context and evoked in time by the phasic stimuli. Direction-specific movement preparatory activity develops during initial learning trials in premotor and oculomotor frontal areas ( learning-dependent changes ) [19,20,27], leading to increasingly specific directional preparatory activity [38] and increasingly accurate and substantial population vectors [39]. In other neurons, movement preparatory activity occurs only transiently during a few initial learning trials ( learning-selective changes ) [20]. These changes in activity reveal a neuronal correlate for the particular role of premotor areas in conditional stimulus response associations and resemble some of the changes seen in the striatum. Neurons in several frontal areas show similar adaptations of existing movement-preparatory and reward-expectation activity to those in striatal neurons in comparable learning tasks [19 21,27,40], although their incidence is lower than in the striatum. Motor activity One of the classical functions of the striatum is related to the preparation and execution of movements. This activity comprises all changes in neuronal activity that are closely related in time to movements. Changes during learning Some neurons in the rostral striatum show preferential movement-related activations while monkeys learn novel movement sequences as an example of procedural motor learning [41]. The activations decrease after the immediate learning phase. By contrast, neurons in the sensorimotor putamen are preferentially activated with learned sequences, suggesting an intra-striatal distribution of activities related to novel versus established performance that is not noticed with other learning changes. In an experiment using single movements, more than half of striatal neurons are activated only when a reward, compared with no reward, is expected. In learning sets, these neurons show also movement-related activations during unrewarded trials, until the differential reward predictions of the novel instruction stimuli become established [18]. Comparable dissociations of movement-sequence learning are found in the supplementary motor area (SMA). More neurons show activations with novel sequences in the pre- SMA than in the SMA proper [42]. In other experiments, neurons in the pre-sma and SMA show relatively little activation with extensively over-trained simple key-press Movement Non-movement Correct Error Correct s s Instruction Trigger Instruction Trigger Fig. 5. Adaptation of movement preparatory activity during learning. With familiar instructions, this caudate neuron showed a sustained instruction response only in rewarded movement trials and a transient response in non-movement trials. During learning, the sustained response increased slightly in movement trials (bottom left) and occurred also in erroneous non-movement trials in which a movement was performed (right). The response in correct non-movement trials resembled that in familiar trials. Trials are plotted chronologically from top to bottom in each raster. Reproduced, with permission, from Ref. [18], q (1998) The American Physiological Society.

6 326 Review Vol.26 No.6 June 2003 movements but they display more activation in animals that have had less training or after behavioral recovery from lesions of the primary motor cortex [43]. Movementrelated activity of prefrontal neurons increases over successive stages of novel learning [23]. During repeated learning episodes, movement-related activity increases gradually in premotor neurons during each episode [19]. These data indicate a role of SMA, premotor and prefrontal neurons in learned movements rather than in control of simple muscle contractions. Long-term changes About one-third of task-related striatal neurons show long-term increases in task-related activity that outlast learning phases of trials and, in some cases, have less selectivity than with familiar stimuli [18]. These changes might reflect consolidation processes of memory storage. The activity increases do not reflect differences in visual stimulus features, as they are reproducible across multiple sets of different visual stimuli and occur even with responses to stimuli that are identical in learning and familiar trials. However, the increases might be related to heightened attention during learning, compatible with the suggested role of attention in learning [9]. The novel instruction stimuli would induce a bottom-up form of attention, and the other task events would also be processed in a less automatic manner during learning. Some striatal neurons show decreased activation outlasting the learning phase, which results in increased task selectivity [18]. Combinations of increases and decreases result in reorganization of task relationships, and some neurons show lesser responses to earlier trial events during learning and greater responses to later events [18]. After habit formation, striatal neurons show less movement-related activity and more responses to movement-initiating stimuli [15]. The decreased responses demonstrate a learning-induced, increased efficacy of neuronal processing, similar to that during visual learning [44]. Neurons in the orbitofrontal cortex show more long-term increases, decreases and reorganization of task-related activity and more changes in task selectivity than do those in the striatum, and their increases occur twice as often as decreases [18]. Neurons in supplementary and frontal eye fields show similar percentages of increases and decreases of activity as in the striatum ( learning-static changes ) [27]. Similar to neurons in the striatum, those in dorsolateral prefrontal cortex show stronger responses to novel stimuli than to familiar ones [40]. They become more finely tuned and selective, and less vulnerable to degradation by noise through learning [44]. Prediction errors The discrepancies between outcomes and their predictions are called prediction errors, in analogy to errors in behavioral performance. Only a few studies have tested the coding of prediction errors in the striatum. TANs respond more frequently to unpredicted rewards than to predicted ones or to movement-triggering stimuli, and they do not respond when rewards are omitted [45], suggesting coding of positive, but not negative, prediction errors. As they are local interneurons bridging striatal striosomes and matrix compartments [46], the TANs might use prediction-error signals to influence other neurons within the striatum. Some neurons in frontal areas respond more to unpredicted rewards than to predicted ones and, thus, appear to code a positive reward-predicting error; other neurons are activated with reward omission during extinction, stimulus response reversal or behavioral errors (negative prediction errors) [47 51]. Neurons activated by behavioral errors, stimulus-movement mismatches or unpredicted rewards are found in dorsolateral prefrontal cortex [52], premotor cortex [53] and the supplementary eye field [54]. With their specific anatomical projections, these neurons might transmit the error signals selectively to groups of postsynaptic neurons for error correction and learning [55]. Synthesis and comments of predictions and expectations Striatal neurons acquire responses to stimuli with behavioral and motivational significance and are sometimes active during the acquisition process. The precise storage mechanisms are unknown, although it is possible, for example, that dopamine responses to unpredicted rewards induce the learning of neuronal responses to reward-related stimuli, on the basis of dopaminedependent synaptic plasticity. Dopamine responses might also induce neuronal activities related to working memory, as suggested by the effects of dopamine agonists and antagonists on striatal and frontal neurons [56,57] and by simulations in neural network models [58,59]. Once the significances of stimuli are learned, stimulus occurrence would activate working memory and evoke expectations of future events. The learning-set experiments demonstrate the presence of movement-preparatory and reward-expectation activity from the first learning trial onwards. Explicit information eliciting these activations cannot be derived from the novel instruction cues. In a different experiment, striatal neurons are also activated in advance of movements and reward, despite the absence of specific stimuli [60]. During learning, a novel stimulus can provide only a point in time for recalling the movement and reward significances from long-term memory of the task context. The memory might have been set up during learning of the basic task structure, possibly via reward-related responses of dopamine neurons. Thus, the movement-preparatory and reward-expectation activity in each initial learning episode can be evoked in time by the stimulus and in content by recall from long-term memory of the task context. As learning advances in each episode, the newly learned instruction would become sufficient to evoke the full information for short-term movement preparation and reward expectation. The activity changes during acute learning phases could reflect this change of evocation mechanisms.

7 Review Vol.26 No.6 June Possible synaptic mechanisms The transient increases in task-related activity might induce synaptic changes at striatal synapses, which could underlie both the learning of stimuli with behavioral and motivational significance and the adaptation of expectations to new contingencies. Increased cortical inputs to striatal neurons might lead to long-lasting synaptic changes, following a Hebbian learning mechanism with conjoint presynaptic and postsynaptic activity. Corticostriatal synapses show LTP or LTD [61,62]. The Hebbian learning mechanism might be supplemented by a three-factor Hebbian process, with a dedicated reward-predicting dopamine signal as determining factor [55]. Dopamine induces plasticity in the striatum and cortex [1,62 64], and interference with dopamine neurotransmission impairs the induction of long-term synaptic changes [61,63,65]. Acknowledgements Our experimental work was supported by the Swiss NSF, the Human Capital and Mobility and Biomed 2 programs of the European Community via the Swiss Office of Education and Science, the Roche Research Foundation, the FRS of Quebec and the NIMH. Writing of this review was supported by the Wellcome Trust. References 1 Reynolds, J.N.J. et al. (2001) A cellular mechanism of reward-related learning. Nature 413, Wise, S.P. (1996) The role of the basal ganglia in procedural memory. Semin. Neurosci 8, Hikosaka, O. et al. (1999) Parallel neural networks for learning sequential procedures. Trends Neurosci. 22, Malamut, B.L. et al. (1984) Monkeys with combined amygdalo hippocampal lesions succeed in object discrimination learning despite 24-hour intervals. Behav. Neurosci. 98, Packard, M.G. and Knowlton, B.J. (2002) and memory functions of the basal ganglia. Annu. Rev. Neurosci. 25, Gaffan, D. and Eacott, M.J. (1995) Visual learning for an auditory secondary reinforcer by macaques is intact after uncinate fascicle section: indirect evidence for the involvement of the corpus striatum. Eur. J. Neurosci. 7, Linden, A. et al. (1990) Slow cortical potentials in Parkinsonian patients during the course of an associative learning task. J. Psychophys. 4, Rescorla, R.A. and Wagner, A.R. (1972) In Classical Conditioning II: Current Research and Theory (Black, A.H. and Prokasy, W.F., eds) pp , Appleton Century Crofts, NY, USA 9 Pearce, J.M. and Hall, G. (1980) A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, Funahashi, S. et al. (1990) Visuospatial coding in primate prefrontal neurons revealed by oculomotor paradigms. J. Neurophysiol. 63, Wilson, F.A.W. et al. (1993) Dissociation of object and spatial processing domains in primate prefrontal cortex. Science 260, Rao, S.C. et al. (1997) Integration of what and where in the primate prefrontal cortex. Science 276, Tremblay, L. and Schultz, W. (1999) Relative reward preference in primate orbitofrontal cortex. Nature 398, Hassani, O.K. et al. (2001) Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. J. Neurophysiol. 85, Jog, M.S. et al. (1999) Building neural representations of habits. Science 286, Aosaki, T. et al. (1994) Responses of tonically active neurons in the primate s striatum undergo systematic changes during behavioral sensorimotor conditioning. J. Neurosci. 14, Aosaki, T. et al. (1994) Effect of the nigrostriatal dopamine system on acquired neural responses in the striatum of behaving monkeys. Science 265, Tremblay, L. et al. (1998) Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol. 80, Mitz, A.R. et al. (1991) -dependent neuronal activity in the premotor cortex: activity during the acquisition of conditional motor associations. J. Neurosci. 11, Chen, L.L. and Wise, S.P. (1995) Neuronal activity in the supplementary eye field during acquisition of conditional oculomotor associations. J. Neurophysiol. 73, Tremblay, L. and Schultz, W. (2000) Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J. Neurophysiol. 83, Brown, V.J. et al. (1995) Responses of cells in the caudate nucleus during visual discrimination learning. J. Neurophysiol. 74, Kubota, K. and Komatsu, H. (1985) Neuron activities of monkey prefrontal cortex during the learning of visual discrimination tasks with go/no-go performances. Neurosci. Res. 3, Germain, L. and Lamarre, Y. (1993) Neuronal activity in the motor and premotor cortices before and after learning the associations between auditory stimuli and motor responses. Brain Res. 611, Schoenbaum, G. et al. (1999) Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J. Neurosci. 19, Niki, H. et al. (1990) In Vision, Memory and the Temporal Lobe (Iwai, E. and Mishkin, M., eds) pp , Elsevier 27 Chen, L.L. and Wise, S.P. (1995) Supplementary eye field contrasted with the frontal eye field during acquisition of conditional oculomotor associations. J. Neurophysiol. 73, Watanabe, M. (1990) Prefrontal unit activity during associative learning in the monkey. Exp. Brain Res. 80, Rolls, E.T. et al. (1996) Orbitofrontal cortex neurons: role in olfactory and visual association learning. J. Neurophysiol. 75, Honig, W. and Thompson, R.K.R. (1982) In The Psychology of and Motivation: Advances in Research and Theory (Bower, G.H., ed.), pp , Academic Press 31 Hikosaka, O. et al. (1989) Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J. Neurophysiol. 61, Alexander, G.E. and Crutcher, M.D. (1990) Neural representations of the target (goal) of visually guided arm movements in three motor areas of the monkey. J. Neurophysiol. 64, Apicella, P. et al. (1992) Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J. Neurophysiol. 68, Watanabe, M. (1996) Reward expectancy in primate prefrontal neurons. Nature 382, Hollerman, J.R. et al. (1998) Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 80, Rainer, G. et al. (1999) Prospective coding for objects in primate prefrontal cortex. J. Neurosci. 19, Lauwereyns, J. et al. (2002) Feature-based anticipation of cues that predict reward in monkey caudate nucleus. Neuron 33, Chen, L.L. and Wise, S.P. (1996) Evolution of directional preferences in the supplementary eye field during acquisition of conditional oculomotor associations. J. Neurosci. 16, Chen, L.L. and Wise, S.P. (1997) Conditional oculomotor learning: population vectors in the supplementary eye field. J. Neurophysiol. 78, Assad, W.F. et al. (1998) Neural activity in the primate prefrontal cortex during associative learning. Neuron 21, Miyachi, S. et al. (2002) Differential activation of monkey striatal neurons in the early and late stages of procedural learning. Exp. Brain Res. 146, Nakamura, K. et al. (1998) Neuronal activity in medial frontal cortex during learning of sequential procedures. J. Neurophysiol. 80, Aizawa, H. et al. (1991) Reorganization of activity in the supplementary motor area associated with motor learning and functional recovery. Exp. Brain Res. 84,

8 328 Review Vol.26 No.6 June Rainer, G. and Miller, E.K. (2000) Effects of visual experience on the representation of objects in the prefrontal cortex. Neuron 27, Apicella, P. et al. (1997) Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioral states. Exp. Brain Res. 116, Kimura, M.et al. (1990) Physiological properties of projection neurons in the monkey striatum to the globus pallidus. Exp. Brain Res. 82, Niki, H. and Watanabe, M. (1979) Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171, Rosenkilde, C.E. et al. (1981) Single cell activity in ventral prefrontal cortex of behaving monkeys. Brain Res. 209, Thorpe, S.J. et al. (1983) The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp. Brain Res. 49, Tremblay, L. and Schultz, W. (2000) Reward-related neuronal activity during go nogo task performance in primate orbitofrontal cortex. J. Neurophysiol. 83, Schall, J.D.et al. (2002) Monitoring and control of action by the frontal lobes. Neuron 36, Watanabe, M. (1989) The appropriateness of behavioral responses coded in post-trial activity of primate prefrontal units. Neurosci. Lett. 101, Flament, D. et al. (1993) Distance- and error-related discharge of cells in premotor cortex of rhesus monkeys. Neurosci. Lett. 153, Stuphorn, V. et al. (2000) Performance monitoring by the supplementary eye field. Nature 408, , Schultz, W. and Dickinson, A. (2000) Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, Williams, S.M. and Goldman-Rakic, P.S. (1995) Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature 376, Inase,M.et al. (1997) Dopaminergic modulation of neuronal activity in the monkey putamen through D1 and D2 receptors during a delayed go/nogo task. Exp. Brain Res. 117, Suri, R. and Schultz, W. (2001) Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13, Dreher, J-C. et al. (2002) A model of prefrontal cortex dopaminergic modulation during the delayed alternation task. J. Cogn. Neurosci 14, Schultz, W. and Romo, R. (1992) Role of primate basal ganglia and frontal cortex in the internal generation of movements: I. Preparatory activity in the anterior striatum. Exp. Brain Res. 91, Calabresi, P. et al. (1992) Coactivation of D1 and D2 dopamine receptors is required for long-term synaptic depression in the striatum. Neurosci. Lett. 142, Reynolds, J.N.J. et al. (2001) A cellular mechanism of reward-related learning. Nature 413, Gurden, H. et al. (2000) Essential role of D1 but not D2 receptors in the NMDA receptor-dependent long-term potentiation at hippocampal prefrontal cortex synapses in vivo. J. Neurosci. 106, Bao, S. et al. (2001) Cortical remodelling induced by activity of ventral tegmental dopamine neurons. Nature 412, Kerr, J.N. and Wickens, J.R. (2001) Dopamine D1/D5 receptor activation is required for long-term potentiation in the rat neostriatum in vitro. J. Neurophysiol. 85, Trends in Neurosciences: a forum for comment Controversial? Thought-provoking? If you wish to comment on articles published in Trends in Neurosciences, or would like to discuss issues of broad interest to neuroscientists, then why not write a Letter to the Editor? Letters should be up to 700 words. Please state clearly whether you wish the letter to be considered for publication. Letters are often sent to the author of the original article for their response, in which case both the letter and reply will be published together. Please note: submission does not guarantee publication.