NEUROSCIENCE UPDATE. KEY WORDS Learning, LTP, Hippocampus, Place fields, Tetrodes, Direction selectivity

NEUROSCIENCE UPDATE Neuronal Dynamics of Predictive Coding MAYANK R. MEHTA Center for Learning & Memory Department of Brain & Cognitive Sciences Massachusetts Institute of Technology Cambridge, Massachusetts A critical task of the central nervous system is to learn causal relationships between stimuli to anticipate events in the future, such as the position of a moving prey or predator. What are the neuronal phenomena underlying anticipation? In this article, I review recent results in hippocampal electrophysiology that shed light on this issue. It is shown that hippocampal spatial receptive fields show large and rapid anticipatory changes in their firing characteristics. These changes are experience- and environment-dependent and can be explained by a computational model based on NMDA-dependent synaptic plasticity during behavior. Striking similarities between the anticipatory network dynamics of widely different neural circuits, such as the hippocampus and primary visual cortex, are discussed. These experimental and theoretical results indicate how the microscopic laws of synaptic plasticity give rise to emergent anticipatory properties of receptive fields and behavior. NEUROSCIENTIST 7(6):490 495, 2001 KEY WORDS Learning, LTP, Hippocampus, Place fields, Tetrodes, Direction selectivity three major issues. First, how perception is transformed, through several cortical processing stages, into an action or motor plan. Second, finding the neuronal representations corresponding to more complex cognitive phenomena such as attention, decision making, and consciousness. And third, understanding the mechanisms responsible for the emergence of neural representations. Although genes contribute significantly toward setting up the overall architecture of the brain, it is believed that a majority of the neural representations are built up by experience-dependent plasticity of synapses. Donald Hebb (1949) proposed a learning rule for the modification of synaptic strengths that can be summarized as follows: If neuron A repeatedly participates in the activation of neuron B, the synapse from A to B is strengthened. Subsequent in vitro experiments showed that synapses, especially glutaminergic synapses, are indeed plastic (Bliss and Lomo 1973). Furthermore, a form of long-term potentiation (LTP) of synaptic strengths that is dependent on the NMDA receptor of glutamate indeed follows a Hebbian learning rule (Levy and Steward 1983; Gustafsson and others 1987; Markram and others 1997; Bi and Poo 1998). If neuron A fires within approximately 50 ms before neuron B, the synapse from A to B is potentiated (LTP), but if neuron A fires within approximately 50 ms after neuron B, the synapse from A to B is de-potentiated (LTD). Neurons in vivo are embedded in vast and complex networks, and it is not clear if the precise temporal activity patterns required for synaptic plasticity in vitro indeed occur in vivo. More important, it is not clear what will be the effect of synaptic plasticity on the activity patterns of complex neuronal networks, and eventually on behavior (Fig. 1). In this article, I review some recent attempts toward understanding these issues. Electrophysiological research during the last several decades has solved an issue that has puzzled humanity for several centuries: How are the properties of the physical world represented in the brain? On a perceptual level, we now know how the firing rates of neurons in a majority of anatomically identified parts of the brain are selectively modulated by physical parameters, such as the color, shape, and velocity of a visual object; the texture of a tactile stimulus; or the intensity and frequency of a sound. We also know the neuronal representation of a majority of bodily movements. Recent research in systems neuroscience has focused on I thank S. Schnall and M. Wilson for a careful reading of the manuscript. This work was supported by the NIH grant #mh58880. Address correspondence to: E18-366, Center for Learning & Memory, Department of Brain & Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139 (e-mail: mayank@mit.edu; web: www.mit.edu/~mayank). Anticipatory Dynamics in CA1 Several physiological and behavioral results suggest that the adult hippocampus may be ideally suited to study the effects of LTP/D on the structure of neuronal networks and behavior. The hippocampus contains a large number of glutaminergic synapses that exhibit NMDAdependent LTP/D. Lesion studies have shown that the hippocampus is critically involved in learning, memory (Scoville and Milner 1957), and spatial navigation (Morris and others 1982). Pyramidal neurons in the rodent hippocampus fire in a spatially selective fashion (O Keefe and Dostrovsky 1971) and provide an accurate estimate of the location of the rat (Wilson and McNaughton 1993). Administration of NMDA antagonists to the hippocampus results in a profound deficit in spatial navigational abilities of a rat (Morris and others 1982), suggesting that the mechanisms of NMDA-dependent LTP/D may indeed be engaged dur- 490 THE NEUROSCIENTIST Neuronal Dynamics of Predictive Coding Copyright 2001 Sage Publications ISSN 1073-8584

Fig. 1. Although much is known at the microscopic level about the biophysical properties of neurons and synapses (~1 micron) and at the macroscopic level about the behavior, little is known about the behavior of large networks (mesoscopic phenomena). Furthermore, a mechanistic understanding of how the microscopic laws of biophysics result in the emergent properties of networks and behavior is one of the central challenges of neuroscience. Fig. 2. The rats first ran along the lower linear track (1.5 m long and 10 cm wide) for food reward at the ends of the track. An overhead camera monitored the position and heading direction of the rat. Each white dot represents the position of the rat at a given moment. Colored dots represent the position of the rat at the time of occurrence of spikes from different pyramidal cells. After running on the first track, for about 30 trials (over about half an hour), the rat was transferred to the second rectangular track, where he again ran about 30 laps. In addition to firing in limited regions of the tracks, called place fields, these CA1 pyramidal neurons were also direction selective, such that they fired robustly when the rat went through the place field in one direction but not the other. ing navigation. Recent studies have shown that transgenic mice that lack NMDA-dependent plasticity in the CA1 region of the hippocampus are severely impaired in spatial naviga- Volume 7, Number 6, 2001 tion tasks (Tsien and others 1996). However, a clear experimental demonstration linking synaptic plasticity to neuronal activity pattern and behavior had been lacking. Hence, we recorded the activity of neurons from the CA1 region of awake behaving rats (Fig. 2) using tetrodes. This method allowed us to record the activity of 120 well-isolated hippocampal neurons simultaneously and hold the same set of neurons for more than 2 months. To detect the effects of synaptic plasticity on the neuronal activity during behavior, we first looked for a change in the overall activity of place cells with experience. Because the behavior of the rat, such as his running velocity or heading direction, are known to strongly influence the firing rate of place cells (McNaughton and others 1983), we first trained the rats to run on narrow linear tracks, such that their behavior during maze running was fairly consistent. Surprisingly, there was a large increase in the firing rate of the place cells within a few (less than 10) traversals of the track, even in these familiar environments (Mehta and McNaughton 1996; Mehta and others 1997). The increased firing rates were maintained during the rest of the experiments (up to 100 more trials over the subsequent hour). Although a change in the firing rate without a change in the environment or behavior is suggestive of a change in the internal dynamics of the brain, it can occur owing to a wide range of phenomena other than synaptic plasticity, such as altered temperature or excitability of the rat. Further analysis revealed that in addition to increasing their firing rates, the centers of these CA1 place fields progressively shifted backward, that is, they shifted in a direction opposite to the direction of movement of the rat. Thus, these CA1 neurons showed anticipatory activity, such that whereas at the beginning of an experiment the neurons encoded the current position of a rat, after experience the neurons encoded the future location of the rat. When the rats were transferred to a second environment after running on the first environment, the place field firing rates were reset to low values. Subsequent repeated traversals of the second environment again resulted in a similar increase in the THE NEUROSCIENTIST 491

firing rate and an anticipatory shift of place fields similar to that seen on the first track. Thus, the anticipatory place field dynamics were environment specific. These results were consistent with neural network models of the effects of NMDA-dependent synaptic plasticity on the recurrent network within CA3 (Levy 1989; Abbott and Blum, 1996; Blum and Abbott 1996; Tsodyks and others 1996; Gerstner and Abbott 1997; Wallenstein and Hasselmo 1997; Kali and Dayan 2000). These network models also showed how the predictive shifts in the place fields can be used to build an experience-dependent cognitive map of the environment, allowing the rat to learn spatial navigational tasks such as the Morris water task (Morris and others 1982). A Feed-Forward Model of Sequence Learning All these models of sequence learning and navigation required the existence of a recurrent excitatory network, such as that in CA3. However, the anticipatory place field dynamics were observed in CA1, which has very few, if any, excitatory recurrent connections. The CA1 pyramidal neurons get their primary excitatory inputs from the entorhinal cortex and CA3. It is conceivable that the observed place field dynamics in CA1 arose in the recurrent network within CA3, and then passively passed onto CA via an unknown mechanism. Such an explanation is neither parsimonious nor sufficient to explain the impaired navigational ability of transgenic mice lacking NMDA-dependent plasticity in only the feedforward connections from CA3 to CA1 (Tsien and others 1996), while the plasticity in the recurrent CA3 network was intact. Hence, we investigated the effects of NMDA-dependent synaptic plasticity on a simple feed-forward network, like the one from CA3 to CA1 (Mehta and others 2000; Mehta and Wilson 2000). The model CA1 neuron initially gets excitatory inputs from a random set of excitatory neurons in CA3 and the entorhinal cortex (Fig. 3). Therefore, the initial synaptic matrix is symmetric on an average. As the rat repeatedly traverses the track, owing to the temporally asymmetric nature of the NMDA-dependent LTP (Markram and others 1997; Bi and Poo 1998), the synapses from those CA3/ entorhinal cells that fire before or after the efferent CA1 neuron will be strengthened or weakened, respectively. Because the amount of NMDA-dependent LTP is larger than LTD, the net result is an increase in the net firing rate of the model CA1 neuron, consistent with previous observations (Mehta and others 1997). Furthermore, the location of the first spike in the CA1 place field undergoes a larger predictive shift owing to LTP, whereas the location of the last spike shows a smaller anticipatory shift due to LTD. Hence, with experience, the model place fields become negatively skewed (Fig. 3b). Consistent with these predictions of the model, it was found that the CA1 place fields indeed showed a large change in their shape. At the beginning of a session, the place fields were symmetric. However, within a few trials, the place fields became highly asymmetric, such that the firing rate of a place cell was low as the rat entered a place field but high as the rat left the place field (Mehta and others 2000). Furthermore, as in the model, the location of the first spike showed a larger anticipatory shift than that of the last spike. The computational model can also explain why place field size and asymmetry are reset when the rat is transferred to a second environment after running on the first. Although the same cell can be active in two different environments, the cell is activated by a different set of inputs, that is, synapses, in the two cases. When the rat is transferred to the second environment, the neurons are activated by different, as yet unpotentiated, synapses. Although the place field dynamics evolved over several minutes and the final values were maintained for at least an hour (during the maze running), these values were reset within seconds of the rat entering the second environment. Thus, the anticipatory place field dynamics are both experience- and environment-specific. Place cells showed anticipatory dynamics even in familiar environments. Perhaps when the rat goes back to his home cage, hippocampal neurons are activated in a different temporal order than during the experiment, resulting in a depotentiation of the synapses. This suggests that the hippocampus stores information about the temporal order in which the rat visited spatial locations within an environment. This hypothesis is further supported by the following experimental observation: The location of a place field of a hippocampal neuron was different when rats traversed a given spatial environment along two different routes (Markus and others 1995). Direction Selectivity from Synaptic Plasticity In addition to the orientation of a bar of light, a majority of neurons in the striate cortex are also strongly selective to the direction of movement of the bar. The above model of sequence learning via NMDA-dependent LTP can shed some light on the origin of direction selectivity, owing to the following similarities between the anticipatory dynamics in the hippocampus and striate cortex. A rat running through the place field in a directional fashion is analogous to a bar of light moving across the retina. Furthermore, visual examination suggests that as in the hippocampus, the firing rates of striate spatio-temporal receptive fields are also asymmetric, such that the rate increases in the preferred direction of motion of the bar (see Livingstone 1998 for a review). It is conceivable that neurons in the striate cortex initially receive random, and hence approximately symmetric, excitatory inputs. During development, small fluctuations in the shape of the synaptic matrix could be amplified by repeated directional activation (caused by natural viewing or spontaneous waves of activity in the retina), resulting in asymmetric excitatory connections in the striate cortex, similar to those in CA1 (Chance and 492 THE NEUROSCIENTIST Neuronal Dynamics of Predictive Coding

others 1998; Mehta and Wilson 2000). A coupling of such asymmetric excitation with delayed inhibition would then result in a large response in the direction of increasing excitation, and low activity in the opposite direction. Thus, the direction selectivity in the striate cortex could arise owing to the same mechanism that generated anticipatory asymmetric place fields. Before experience After experience Fig. 3. A, A computational model of the effect of NMDA-dependent plasticity on a simple feed-forward network. The strength of the connections between the afferent CA3 (and entorhinal cortex) pyramidal neurons and CA1 neurons is proportional to the thickness of the lines. Yellow lines indicate very low (~ 0) synaptic weights. The rat travels through the place field from left to right (white arrow). Before experience (red lines), the synaptic matrix is symmetric. After experience, the synaptic matrix becomes asymmetric (cyan lines). B, The firing rate of the resultant place fields after experience (cyan) differ from that before (red) as follows (arrow). The first shows a large anticipatory shift due to long-term potentiation (LTP), the location of the last spike in the place field shifts back due to long-term de-potentiation (LTD), and the mean firing rate increases. As a result, the peak firing rate in the place field and the location of the peak (or the center of the place field) also show anticipatory shifts. In general, temporally ordered activation of neurons can result in anticipatory dynamics and direction selectivity in any abstract parameter space. Indeed, anticipatory dynamics have been observed in several cortical regions. Neural activity of the supplementary eye field during the period following the animal s response, and before the reward, appeared earlier with experience (Chen and Wise 1995). Also, neural activity in the primate prefrontal cortex, corresponding to the direction of the animal s impending responses, progressively appeared earlier with experience (Asaad and others 1998). Not only could these seemingly different phenomena in widely different parts of the brain have a common origin, namely, temporally asymmetric synaptic plasticity during learning, they perhaps serve a similar cognitive function: to learn the causal relationships between stimuli, thereby enabling the animal to anticipate events in the future, such as the future location of itself or an object moving across the visual field, based on past experience. Future Directions Hippocampal neuronal activity is strongly modulated by an 8-Hz theta rhythm during active exploration. It has been suggested that such rhythmic activity may play a critical role in learning and memory. This hypothesis was strengthened by a recent observation that the phase of the theta rhythm, at which a place cell fires a spike, steadily advances to lower values as a function of the position of the animal (O Keefe and Recce 1993). Thus, the phase of the theta rhythm contains information about the location of the animal. The mechanism by which such a placedependent phase code can arise is still being debated (Jensen and Lisman 1996; Tsodyks and others 1996; Wallenstein and Hasselmo 1997; Kamondi and others 1998; Bose and others 2000). Our preliminary analysis suggests that an interaction between asymmetric excitation, resulting from NMDA-dependent plasticity and theta rhythmic oscillations, could result in a systematic variation of spike timing within a place field (Mehta and others 2000). For example, when excitation is low, the neuron would take a long time to come out of the periodic inhibition and fire a spike, resulting in a large phase. Similarly, when the input is high, the neuron would come out of inhibition earlier, resulting in a low-phase spike. We have shown that the firing rate in the place field is Volume 7, Number 6, 2001 THE NEUROSCIENTIST 493

low at the beginning of a place field and high at the end. This would result in the observed high phase at the beginning of the place field and low phase at the end. Thus, the place field asymmetry could be the origin of phase precession. A similar phenomenon, that of reduction in latency to spiking in the preferred direction, has been observed in the spatiotemporal receptive fields in striate cortex as well (Livingstone 1998), suggesting that a mechanism similar to phase precession may be involved. Further work along this direction could provide an insight into the nature of the neuronal code. In addition to Hebbian, that is, spike-timing-dependent, plasticity, hippocampal (and cortical) neurons also exhibit other types of plasticity in vitro (see Bear and Malenka 1994 for a review). For example, high-frequency stimulation produces LTP but low-frequency stimuli elicit LTD. Does frequency-dependent plasticity occur in vivo? If yes, what is its effect on neuronal networks and behavior? Do various forms of synaptic plasticity interact in any way? Our preliminary work suggests that, even though the firing rate in a place field is high at the end and low at the beginning, higher-frequency (theta burst) firing occurs at the beginning of a place field but lowerfrequency spikes occur at the end of a place field. This could result in a frequency-dependent LTP at the beginning of a place field and LTD at the end, thereby resulting in an enhanced anticipatory dynamics at the single cell level. Finally, dopaminergic neurons exhibit a shift from the conditioning stimulus to the unconditioned stimulus after training, thus encoding important information about reward contingency (Shultz and others 1997). Although several experimental and theoretical works have investigated the effects of such a reinforcement signal, little is known about the origin of this rewardpredictive shift in the dopaminergic neurons activity. We suggest that a likely mechanism is the NMDAdependent association between sequential presentations of stimulus and reward, similar to the anticipatory mechanism in CA1. Thus, an interaction between the reward contingency and plasticity could enhance the learning of goal locations. Recent results support such an interaction: there are a larger number of place fields near the goal locations (Hollup and others 2001). In addition to normal functioning, synaptic plasticity may play a critical role in neuropathology. It has been suggested that focal epilepsy could occur via formation of runaway associations between memories (Mehta and others 1993) owing to Hebbian learning resulting from electrical or chemical perturbations. A better understanding of the dynamics of sequence learning could suggest novel cognitive and pharmacological ways to tackle epilepsy. Thus, NMDA-dependent plasticity could be a universal mechanism of anticipatory learning, which when coupled with other mechanisms could result in a diverse set of complex cognitive abilities. References Abbott LF, Blum KI. 1996. Functional significance of long-term potentiation for sequence learning and prediction. Cereb Cortex 6:406 16. Asaad WF, Rainer G, Miller EK. 1998. Neural activity in the primate prefrontal cortex during associative learning. Neuron 21:1399 407. Bear MF, Malenka RC. 1994. Synaptic plasticity: LTP and LTD. Curr Opin Neurobiol 4:389 99. Bi G-q, Poo M-m. 1998. Activity-induced synaptic modifications in hippocampal culture: dependence on spike timing, synaptic strength and cell type. J Neurosci 18:10464 72. Bliss TV, Lomo T. 1973. Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. J Physiol 232(2):331 56. Blum KI, Abbott LF. 1996. A model of spatial map formation in the hippocampus of the rat. Neural Comp 8:85 93. Bose A, Booth V, Recce M. 2000. A temporal mechanism for generating the phase precession of hippocampal place cells. J Comp Neurosci 9(1):5 30. Chance FS, Nelson SB, Abbott LF. 1998. Synaptic depression and the temporal response characteristics of V1 cells. J Neurosci 18:4785 99. Chen LL, Wise SP. 1995. Neuronal activity in the supplementary eye field during acquisition of conditional occulomotor associations. J Neurophysiol 73:1101 21. Gerstner W, Abbott LF. 1997. Learning navigational maps through potentiation and modulation of hippocampal place cells. J Comp Neurosci 4:79 94. Gustafsson B, Wigstrom H, Abraham WC, Huang Y-Y. 1987. Long-term potentiation in the hippocampus using depolarizing current pulses as the conditioning stimulus to single volley synaptic potentials. J Neurosci 7:774 80. Hebb DO. 1949. Organization of behavior: a neuropsychological theory. New York: John Wiley. Hollup SA, Molden S, Donnett JG, Moser M- B, Moser EI. 2001. Accumulation of hippocampal place fields at the goal location in an annular watermaze task. J Neurosci 21:1635 44. Jensen O, Lisman JE. 1996. Hippocampal CA3 region predicts memory sequences: accounting for the phase precession of place cells. Learn Mem 3(2 3):279 87. Kali S, Dayan P. 2000. The involvement of recurrent connections in area CA3 in establishing the properties of place fields: a model. J Neurosci 19:7463 77. Kamondi A, Acsady L, Wang XJ, Buzsaki G. 1998. Theta oscillations in somata and dendrites of hippocampal pyramidal cells in vivo: activity-dependent phaseprecession of action potentials. Hippocampus 8(3):244 61. Levy, WB. 1989. Computational models of learning in simple neural systems. In: Hawkins RD, Bower GH, editors. Computational neuroscience: trends in research. New York: Academic Press. p 243 305. Levy WB, Steward D. 1983. Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus. Neuroscience 8:791 7. Livingstone MS. 1998. Mechanisms of direction selectivity in macaque V1. Neuron 20(3):509 26. Markram H, Lubke J, Frotscher M, Sakmann B. 1997. Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275:213 15. Markus EJ, Qin YL, Leonard B, Skaggs WE, McNaughton BL, Barnes CA. 1995. Interactions between location and task affect the spatial and directional firing of hippocampal neurons. J Neurosci 11: 7079 94. McNaughton BL, Barnes CA, O Keefe J. 1983. The contributions of position, direction, and velocity to single unit activity in the hippocampus of freely-moving rats. Exp Brain Res 1:41 9. Mehta MR, Barnes CA, McNaughton BL. 1997. Experience-dependent, asymmetric expansion of hippocampal place fields. Proc Natl Acad Sci U S A 94(16):8918 21. Mehta MR, Dasgupta C, Ullal GR. 1993. A neural network model for kindling of focal epilepsy: basic mechanism. Biol Cybern 68(4):335 40. Mehta MR, McNaughton BL. 1996. Expansion and shift of hippocampal place fields: evidence for synaptic potentiation during behavior. In: Bower J, editor. 494 THE NEUROSCIENTIST Neuronal Dynamics of Predictive Coding

Computational neuroscience: trends in research. New York: Plenum. p 741 5. Mehta MR, Quirk MC, Wilson MA. 2000. Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron. 25(3):707 15. Mehta MR, Wilson MA. 2000. From hippocampus to V1: effect of LTP on spatiotemporal dynamics of receptive fields. Neurocomputing 32:905 11. Morris RG, Garrud P, Rawlins JN, O Keefe J. 1982. Place navigation impaired in rats with hippocampal lesions. Nature 297(5868):681 3. O Keefe J, Dostrovsky J. 1971. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res 34(1):171 5. O Keefe J, Recce ML. 1993. Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus 3(3):317 30. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science 275:1593 9. Scoville WB, Milner B. 1957. Loss of recent memory after bilateral hippocampal lesions. J Neuropsychiatry Clin Neurosci 12(1):103 13. Tsien JZ, Huerta PT, Tonegawa S. 1996. The essential role of hippocampal CA1 NMDA receptor-dependent synaptic plasticity in spatial memory. Cell 87(7):1327 38. Tsodyks MV, Skaggs WE, Sejnowski TJ, McNaughton BL. 1996. Population dynamics and theta rhythm phase precession of hippocampal place cell firing: a spiking neuron model. Hippocampus 6(3):271 80. Wallenstein GV, Hasselmo ME. 1997. GABAergic modulation of hippocampal population activity: sequence learning, place field development, and the phase precession effect. J Neurophysiol 78(1):393 408. Wilson MA, McNaughton BL. 1993. Dynamics of the hippocampal ensemble code for space. Science 20:261(5124): 1055 8. Volume 7, Number 6, 2001 THE NEUROSCIENTIST 495