An Associative Neural Network Model of Classical Conditioning

Transcription

1 Department of Numerical Analysis and Computer Science TRITA-NA-P27 ISSN -225 ISRN KTH/NA/P-2/7SE An Associative Neural Network Model of Classical Conditioning Christopher Johansson and Anders Lansner Report from Studies of Artificial Neural Systems, (SANS)

2 Numerical Analysis and Computer Science (Nada) Royal Institute of Technology (KTH) S- 44 STOCKHOLM, Sweden An Associative Neural Network Model of Classical Conditioning Christopher Johansson * and Anders Lansner TRITA-NA-P27 Abstract In this paper we present a new associative model of classical conditioning based on a neural network. The new model is compared with a number of other well-known models of classical conditioning. The experiments that are used to evaluate the new model are commonly used and they represent the set of tasks that a model of classical conditioning needs to address in order to be successful. The new neural network based model is composed of a number of interconnected Bayesian confidence propagating neural networks (BCPNNs). The BCPNN implements Hebbian learning. This new BCPNN based model sorts under the category of associative models. A key concept of this model is to make a closer tie between the output and the underlying neural activity. The BCPNN model does not use delaylines as many other models of conditioning. The output from the BCPNN model fit the results of classical conditioning experiments. Keywords: Classical Conditioning; BCPNN; Neural Network; Learning; Associative Model; Hebbian Learning; * cjo@nada.kth.se

3 Introduction Classical conditioning is when an animal learns to associate a stimulus with a reinforcement or aversion. The study of conditioning goes back to the beginning of the 2 th centaury and the experiments with dogs performed by the Russian psychologist Pavlov. When dogs are presented with food, which is an unconditioned stimulus,, they start to salivate and this response is called the unconditioned response, UR. In Pavlov s experiment the dogs heard a tone (conditioned stimulus, CS) before they were presented with the food. After a number of training trials the dogs were able to pair the CS with the and started to salivate when only the CS was presented (conditioned response, CR). If the training is continued, after the initial learning period, the intensity or probability of a response is increased. The distinction between UR and CR is less relevant in many parts of this text and hence we only use the term response, R. The learning processes involved in classical conditioning have been studied for a long and many models of these processes have been developed (for a review see Ph.D. thesis by J. Morén []). Classical conditioning has attracted model-makers since there exists lots of data on the phenomenon and the basic workings of conditioning are easy to understand. But conditioning is also a very difficult subject to study in the sense that the underlying biological system that is studied, i.e. the animal, is a very complex system. Small variations or changes in the experimental settings may have a large impact on the result. The type of sensory modality used for the conditioning stimulus plays an important role, e.g. animals cannot learn to associate a flavour with a shock. There is a general consensus that what we call learning is an intricate process of both explicit and implicit nature and many of these learning processes are closely connected to and affected by emotions. This means that the effects seen in classical conditioning experiments do not originate from a single learning system but from several different interconnected learning mechanisms. A conclusion from this is that in order to be able to describe all aspects of conditioning experiments one needs to have a model of considerable complexity, a single differential equation will not do, e.g. when several stimuli are present and the desired response depends on their relative activity. The models used for describing the processes of classical conditioning are dived into two classes; computational models and associative models. Roughly speaking, a computational model can be said to be a black box with statistical computations used to analyse and predict future data. In this paper we are concerned with associative models and not computational models. Associative models are based on the idea that the inner working of the biological system that is modelled is based on associations of events. This means that the model cannot only be used for predicting output data but also for predicting how this data was created. Computational models on the other hand do not consider how the output data was created. They are only concerned with predicting the output data. A more comprehensive distinction between these two types of models is provided by Gallistel [2].

4 There are two types of associative models; trace-based models like Sutton and Barto s and neural network based models. The trace-based models are not able to predict the inner workings of the biological system they model. Many of the neural network based models uses delay lines to handle the aspects of, i.e. create the correct dynamics of the response. An issue with delay lines is that they introduce a large amount of hardwiring into a model. The new model we present in this paper is a fusion between these two types, trace and neural network based, of associative models. From this fusion we get a more biological realistic model and at the same it retains the capabilities possessed by these other models. In this paper we aim at making a model with a simple phenotype but with a rich set of behaviours. The model is built with fairly simple building blocks; an advanced model of the perceptron. But creating an overall spanning model of classical conditioning is also hard since there is a wide variety of conditioning experiments, which often gives contradicting results. Traditionally classical conditioning has been an area studied by psychologists and many of the models of classical conditioning lack neural correlates. A key issue in the new model presented in this paper is to identify the correlation between the neural activity in the model and the results seen in the experiments. The model we present is built with a number of interconnected Bayesian confidence propagating neural networks (BCPNNs) [3-6]. The BCPNN has been used to model a variety of different systems and phenomena [7-9]. Classical Conditioning Experiments In the following sections a number of typical experiments used to test classical conditioning models are presented. The outline and description of the experiments follows closely that of Balkenius []. In many texts about learning and acquisition the figures measures on the x- axis. The is measured in units of varying size, from seconds to days, and the meaning of is very general e.g. it could mean number of trials. In this paper always means a measure in the order of magnitude of seconds. The speed of learning varies largely between different experiments i.e. associations to food can be learnt after a single trial while associations to an air puff in the eye may take 5 trials or more to learn. In most of the experiments in this paper we assume that learning is complete after one trial except for the experiments on S- shaped learning curves and reacquisition. We use one trial learning because it enables the display of the dynamics of the activity during the conditioning experiments. With the explicit display of the activity it is relative easy to grasp how the, CS and R interact. Acquisition and Learning Curve The most important process in classical conditioning is acquisition, where an association between a stimulus and response is made. Pavlov s experiment with dogs, described in the introduction, is a typical example of an acquisition process. CS + CS R

5 First a CS is presented; it is then followed by a. This process is repeated a number of s and as a result a CS that is later presented on its own will be able to produce R. Delay Conditioning A ISI CS LCS L Delay Conditioning B ISI CS LCS L Trace Conditioning ISI CS LCS L Figure Three different types of training procedures. There are two versions of delay conditioning; A the CS disappears immediately before the is presented. B the CS is still present when the is presented. In trace conditioning there is a silent period before the is presented. The ISI interval is computed as the difference between the start of the CS and. In this paper we assume that the learning is complete after one trial but in reality it often takes more than one trial for an animal to acquire an association. When the learning process occurs over multiple trials a desirable property of a model is to show an S-shaped learning curve.

6 Inter-Stimulus Interval Effects The inter-stimulus interval (ISI) is central in classical conditioning. The ISI period measures the between the start of CS-stimulus and -stimulus. The basic acquisition process can be performed in three different setups (Figure ); Delay A conditioning where the CS ends immediately before the onset of, Delay B conditioning where the does not end until the ends and Trace conditioning where there is a silent period between presentation of the CS and the. Trace conditioning is different from delay conditioning in that the CS needs to be memorised. In many of the models we present in this paper the memory of the CS is formed by a decaying trace. In trace conditioning, the ISI can be both positive and negative. If the appears before the CS is presented the association becomes negative i.e. an inhibitory connection is formed. The desirable behavior for the response level to have a single peak at small positive ISI:s, no response at all for negative ISI:s and an asymptotically declining value as the ISI grows large. Extinction Extinction is when an acquired association between CS and R is erased or suppressed. The ability to relearn and extinct previous made associations is often important in order to adapt to changes in the environment. Extinction of a previously made association is simply done by presenting the CS on its own with out any following. CS CS no R Reacquisition Effects Reacquisition effects appear when an animal relearns a previously extinguished association. The acquisition is faster during the succeeding trials after the initial acquisition. Here, running an acquisition-extinction cycle four s tested the reacquisition effect. Somes the term savings effects is used instead of reacquisition. Blocking Blocking means that when a first stimulus, CS, has been conditioned, i.e. associated with a response, a second stimulus, CS 2, will be blocked from being associated with the response, R. The blocking experiment shows that acquisition of an association is not independent of earlier learning. In blocking no learning occurs when the second stimulus, CS 2, is presented since CS has already been associated with R. CS + CS R CS CS CS R no In the reacquisition experiment one is looking for the remains of the previously suppressed association. The remains of the first learning trial have the effect of speeding up the learning in the following trials.

7 Conditioned Inhibition In conditioned inhibition two stimuli, CS and CS 2 are conditioned on R. A third stimulus, CS, is then presented together with one of the previously conditioned stimuli, CS, and no is given. The result is that the CS takes on inhibitory properties, which has as a result that if CS is presented together with CS 2 there will be no response, R. Phase I CS2 + Phase II CS + CS + CS+ no Test CS + CS no R 2 Conditioned inhibition is an active form of extinction and it requires the existence of inhibitory associations between the conditioned stimulus and response. Secondary Conditioning Secondary conditioning is when a conditioned stimulus CS is used to condition a second stimulus CS 2 on the response, R. The effect is typically weak and highly dependent on the exact timing of CS and CS 2, as CS will extinguish at the same that CS 2 is reinforced. CS+ CS R CS + CS CS R 2 2 Secondary condition is by some considered to be an important part of a model of instrumental learning. Models of secondary conditioning or instrumental learning have been used to solve path-finding problems []. But secondary conditioning need not play a role in instrumental learning since there are neural models [2] that can solve the path-finding problem by other means than secondary conditioning. Facilitation by an Intermittent Stimulus A second type of acquisition can be seen in trace conditioning when an extra stimulus CS 2 is introduced during the silent period between the presentation of CS and. If the conditioning to CS is weak due to a long ISI the extra CS 2 will facilitate conditioning to CS. Normal CS+ CS weak R Facilitated CS + CS + CS strong R 2 Models of Classical Conditioning In the paper by Balkenius [] five different models are analysed; the Sutton-Barto, the Temporal-Difference, the Klopf, the Balkenius and the Schmajuk-DiCarlo model. These five models have a roughly equal complexity and they are also making the same claims about their capabilities. All of these models are associative models. The

8 Klopf and Balkenius models have been influenced by neural network ideas. In all of these models the learning is initiated by changes in the input/output values. The Sutton-Barto (SB) model is built around a single, first order, differential equation and it works in real-. The model is an extension of a previous model by Rescorla-Wagner, which did not work in real-. The SB model is over 2 years old and it has been a precursor of many later associative models. The Temporal-Difference (TD) model is an extension of the SB model where the concept of discounted rewards has been introduced. The Klopf model incorporates some basic ideas from neural networks and the equation describing the model could also be used to the describe the relationship between units in a feed-forward neural network with delay lines. The output, R, is computed by weighting the incoming CS(s) both in and for each stimulus. The values of the weights are changed by training with a set of constants (the delaylines). Excitatory and inhibitory training is separated and there are both negative and positive weights. Neural networks have also influenced the Balkenius model and as the Klopf model it also uses delay-lines. The separate training of excitatory and inhibitory connections is even more distinguished than in the Klopf model. The equation of the weight computation is very similar to the differential equation used in the Sutton-Barto model. The Balkenius model can be seen as an attempt to combine TD-learning and neural networks. The Schmajuk-DiCarlo (SD) model is based on differential equations and the concept of short-term memory is modelled by a trace in a fashion similar to that used in the BCPNN-model. Bayesian Confidence Propagating Neural Network The Bayesian confidence propagating neural network (BCPNN) algorithm is derived from Bayesian statistics [3]. The weights are computed according to Hebb s principle [4] of strengthening co-activated units. A population is defined as a set of units and a projection as the computation needed to derive the weights and biases of a connection between two populations. A projection has a direction and the weights and biases are included in it. A very important feature of the BCPNN is the partitioning of units into hypercolumns, but in this paper we will only be concerned with single hypercolumn populations. More information about hypercolumns is found in the references on the BCPNN [3, 5, 3]. In the following equations N is used to denote the total number of units in a population and H (which always equals ) as the number of hypercolumns in a population. In a population with k hypercolumns, each hypercolumn will have U k units, which gives N= k U k. The differential equations are preferably solved by Euler s method since it is the simplest method available. The first two equations, eq. () and (2), represents the synaptic potential in the preand postsynaptic synapses. The build-up of this potential is controlled by the parameters τ Zpre and τ Zpost. The inputs to the pre- and postsynaptic units are represented by the variables S i and S j. If τ Z the synaptic potential is instantaneously built up but it disappears when the input disappears. If τ Z is small,

9 slightly larger than, the synaptic potential builds up and disappears rapidly and if τ Z is large the build up and decay of the synaptic potential is slow. Zi( t) Si( t) Zi( t) = () t τ Zpre Z j( t) Sj( t) Z j( t) = (2) t τ The second set of equations, eq. (3)-(5), represent the synaptic trace in a connection between two units. The function of these three equations is to delay the storage of correlations. The synaptic trace, stored in the E-variables, is thought to correspond to the Ca 2+ influx in a synapse after it has been activated and which is necessary for synaptic potentiation [5]. If τ E then there is no synaptic trace, the activity of Z i and Z j is instantaneously propagated to the memory in eq. (6)-(8). If τ E is small, slightly larger than, the synaptic trace is short and if τ E is large the synaptic trace is consequently long. Ei( t) Zi( t) Ei( t) = (3) t τ E Ej( t) Z j( t) Ej( t) = (4) t τ E Eij ( t) Zi ( t) Z j ( t) Eij ( t) = (5) t τ In the following three equations, eq. (6)-(8), the P-variables constitute the memory of a connection and are thus intended to correspond to the long-term potentiation (LTP) in a synaptic coupling [5]. When the printnow-signal is activated, i.e. the variable κ is greater than zero, the information stored in the state of the synaptic coupling, E-variables, is transferred into the memory (P-variables) by eq. (6)-(8). The activation of the printnow-signal is thought to correspond to the release of intercellular neuromodulator substances [6] e.g. dopamine [7]. The printnow-signal is in the range of (, ) and a large printnow-signal induces a large change in the memory. If τ P is small the memory is short-term and only a few patterns or correlations are remembered and if τ P is large the memory is long-term. Pt i( ) Ei( t) Pt i( ) = κ (6) t τ P Pj( t) Ej( t) Pj( t) = κ (7) t τ P Pij ( t) Eij ( t) Pij ( t) = κ (8) t τ At each -step the P-variables are used to compute the weights and biases. The constant λ is used to avoid the logarithm of zero and it can be thought of as the bias activity or as the noise in a synapse. Zpost E P

10 βi() t = Pi() t + λ (9) 2 Pij () t + λ wij () t = ( Pt () + λ )( P() t+ λ ) () i j The retrieval process is divided into two parts, first the potential is computed then the new activity. The potential h i is computed by integrating over the support given from the incoming connections. Before the retrieval process is started the potential is initialised to zero. The potential is then integrated, by eq. (), until it settles for a stable set of values. The constant τ C is the potential integration factor or in more neuropsychological terms the membrane constant. It determines how fast the potential of a unit adapts to new input. We define a set C(k) that consists of all units, j, in a hypercolumn k. In the following two equations we use the set C(k) to describe how the new activity, o i, is computed. The new activity, o i, is computed by normalising the support in each hypercolumn. hi () t = log ( β ()) + log () () () / () t for each hypercolumn : ( ) = (2) H i t wij t oj t hi t τ C k j C( k) Ghi () t e k oi t N Ghj ( t) e j C( k) The characteristic of the BCPNN is determined by the values of τ Zpre, τ Zpost, τ E, τ P, τ C and G. The values of the four τ parameters are in the range of (, ) and the G parameter is in the range of [, ). The gain, G, is used to control the shape of the Gibbs distribution used to compute the output activity from a population. The differential equations were solved with Euler s method and the -step, t, was set to. In the implementation the lower bound of the parameters τ then become t instead of. If two populations are of equal size a non-plastic one-to-one projection can be created between these populations. The one-to-one projection connects unit n in the first population with unit n in the second population and so on. A one-to-one projections is created with the following settings; β=λ and w=/ λ. A population have two modes of operation; silent and active. When there is no activity in a population i.e. the activity is zero for each unit, we say that the population is silent. A silent population does not affect any other population. If there is activity in population i.e. all units have bias activity, we say that the population is active. If the units in a population have bias activity they do affect other populations because of the bias β.

11 Method The Neural Model The neural network based model we used had three population; CS-, - and R- population. The CS- and -populations were used to represent incoming stimuli and they were both connected with the R-population (Figure 2). The two connections between the CS-population and the R-population were plastic while the connection between the -population and the R-population was a non-plastic direct projection. CS-population unit CS stimuli unit 2 inhibitory and excitatory all-to-all plastic projection R-population unit 9 unit unit 2 unit 2 -population unit unit 9 stimuli unit 2 non-plastic one-to-one projection unit 2 unit 9 unit 2 Figure 2 The BCPNN model of classical conditioning. Three populations were used; each represented an input or output node of the system e.g. a CS input was set in the CS-population. There were two plastic projections from the CS-population to the R-population, one was inhibitory and the other was excitatory. The projection between the -population and the R- population was static. The model had 2 units in each population. At least 2 units should be used in each population in order to attain a low bias level of activity, as the activity was normalised within each population 2. If e.g. two units had been used, the bias level of activity would have been 5% or.5, with twenty units the bias level of activity become 5% or.5. 2 In this special case there was only one hypercolumn in the population and therefore we normalised over the entire population.

12 The non-plastic connection had the property that if unit n in the -population was active then unit n in the R-population was heftily excitated. This meant that by setting a unit active in the -population a corresponding unit also became active in the R- population. One of the two plastic projections was excitatory and the other was inhibitory. Which of the two projections that was updated was determined by the printnowsignal. If the printnow-signal was positive then the excitatory projection was updated and if the printnow-signal was negative then the inhibitory projection was updated. If only one stimulus was present the activity of this stimulus / unit was set to - (N- )λ and the remaining stimuli / units were set to λ. When two stimuli were present the activity of each active stimulus was set to - (N-2)λ and the remaining stimuli were set to λ. This meant that when two stimuli were present the activity of each active stimulus was about.5. A central part in all models of classical conditioning is the triggering of learning, i.e. detecting that new information is available and that it should be learnt. In all of the models mentioned the rate of learning is an artifact of the past and current level of change in the CS and inputs. In the BCPNN-model the printnow-signal controls the learning rate. The printnow-signal was computed according to eq. (3) where o was the activity in one of the units in the R-population and k was a scale factor. In all experiments except one k=5. We assumed a small noise level of at least 2% in the activity and therefore changes smaller than.2 in the derivative of the activity was neglected. In the implementation, the.2 threshold enabled us to avoid boundary problems between the different phases of input to the model. The printnow-signal was gated by the. If no was present the printnowsignal and hence the learning-rate was zero. This can be motivated by direction of attention; if no attention is paid to the stimulus then it has no relevance and hence should not affect the learning. k o( t) if o( t) >.2 printnow - signal = (3) t t else In all of the models, except the Klopf model, there exists a trace of the CS-input. In the Klopf model the trace of the CS-input is replaced by a summation over delay lines. The corresponding trace in the BCPNN-model is found in the Z-variables. The -dynamics of the response was controlled by the membrane potential, eq. (), in the BCPNN-model while in the Klopf and Balkenius models the response dynamics is created by the delay-lines. The following parameter settings were used in the model; τ Zpre =, τ Zpost =, τ E =, τ P =2, τ C =5 and λ =.. All populations except the R-population had G=, the R- population had G=3.

13 Results The results of the BCPNN model applied to the previously described experiments are grouped in four sections. The given on the x-axis is intended to be in the same order of magnitude as seconds. When an acquisition was repeated the label on the x- axis was changed from to number of trials. All of the plots follow the same layout; the first graph shows the printnow-signal, the second graph shows the response, the third (and following) graph(s) show the CS and the graph at the bottom shows the. As mentioned earlier the printnow-signal controlled the learning rate of the CS-to-R projections. A positive printnow-signal meant that the excitatory projection was updated and a negative printnow-signal meant that the inhibitory projection was updated Number of Trials Figure 3 The learning curve over 5 training trials. The printnow-signal was set to % of its normal strength. Trace conditioning with an ISI of 4 units was used. The learning curve was very slightly S-shaped. Acquisition and Inter-Stimulus Interval First we studied the effects of repeated learning and the form of the learning curve (Figure 3). In order to avoid instantaneous learning the printnow-signal was set to % of its original strength (eq. (3)). We used trace conditioning and run 5 training trials and the result was a slightly S-shaped learning curve. The length of the inter-stimulus interval (ISI) affects the acquisition of a stimulusresponse association (Figure 3). Three different types of acquisition were considered

14 and delay B conditioning gave the highest probability / strongest response. Delay A and trace conditioning were both identical for small ISI-periods, less than 2 units. The sharp rise of the response seen for delay A and trace conditioning was due to the build-up of the potential, the had to be present for at least -units in order to induce a potential in the R-population. When the ISI-period was larger than 2 -units the effects of trace conditioning started to become apparent with a lower level of response as a consequence. If the ISI period was 8 -units long the trace or memory of an input had almost completely disappeared. Delay A and B conditioning had almost identical response levels when the ISI was larger than 2 -units. Delay A Delay B Trace.8 Probability ISI () Figure 4 The inter-stimulus interval (ISI) effect in delay and trace conditioning experiments. During the first 2 -units the delay A and trace conditioning curves were almost identical and later the curves of delay A and B conditioning were identical. The sharp rise of the response during the first -units seen in delay A and trace conditioning occurred because of the buildup of potential in the R-population. In Figure 5, Figure 6 and Figure 7 a detailed view of the acquisition process is presented. The ISI-period was set to 4. The strongest response was achieved when delay A or B conditioning was used. Trace conditioning produced a slightly weaker response. The initial 7 -units in Figure 5, Figure 6 and Figure 7 were used to acquire the association. These three figures illustrate the three different acquisition processes described in Figure. Notice how the changed level of response results in a printnowsignal. After the acquisition of the association there was a silent period of 4 units. The silent period allowed the activity in the network to go down towards its bias level. At 2 the CS was given to test the strength of the acquired association. The length of this test period was 3 -units.

15 Print CS Figure 5 Delay A conditioning with a ISI-period of 4. The CS ended before the input. The first 7 units were used to acquire the association and the remaining 8 units were used to test the association. Print CS Figure 6 Delay B conditioning with a ISI-period of 4. The CS did not end at the onset of the but ended at the same as the. The first 7 units were used to acquire the association and the remaining 8 units were used to test the association. Extinction and Reacquisition The extinction experiment in Figure 8 started with the acquisition of an association during the first 5 units. Between the 5 and 5 there was a silent period.

16 At the 9 the extinction process was initiated. The extinction process worked almost as the acquisition process with the only difference that the was not presented 3. As a result the response level, when tested again at 25, was much lower than previously (at 8). Notice that a negative printnow-signal was created because the was absent. The reacquisition effects were tested by four consecutive acquisition-extinction cycles (Figure 9). After the first acquisition-extinction cycle the response was harder to facilitate, the complete opposite of the desired behavior where the response is more easily facilitated in the following cycles. Print CS Figure 7 Trace conditioning with a ISI-period of 4. CS ended long before the became active. The first 7 units were used to acquire the association and the remaining 8 units were used to test the association. Blocking Figure shows the effect of blocking. First an association is acquired for CS. This was followed by a silent period of 5 -units. Between and 7 a new acquisition was initiated and a second stimulus, CS 2, was introduced. During this new acquisition process the printnow-signal was zero. The printnow-signal was zero because the first stimulus, CS, already predicted a % response and no change in the level of response occurred when the second stimulus, CS 2, was introduced. Hence when the was activated the response was already at % and therefore no printnow-signal was produced. During the last 5 units the response given by the two stimuli was tested. The response level of CS remained high and the response level of CS 2 remained low. 3 The very small activity of the was the bias activity. The bias activity was used to indicate that something was expected to happen.

17 Print CS Figure 8 Extinction of an acquired response. The response given by the CS was almost at 8. After the extinction process at 25 the level of response following a CS was reduced to about Number of Trials Figure 9 The reacquisition effects were tested by running four acquisition-extinction cycles. After the first cycle the response was harder to facilitate. The printnow-signal was set to % of its normal strength.

18 Print CS CS Figure Blocking of a second stimulus. An association is first acquired to CS. Then both CS and CS 2 were conditioned on the. This did not associate the CS 2 with the response. Print CS CS CS Figure Conditioned inhibition. Activity in CS or CS 3 had an excitatory influence on the response. At 25 to 28 the CS 2 was trained to have an inhibitory effect on the response.

19 Conditioned Inhibition Conditioned inhibition shows that a stimulus can attain an inhibitory influence on the response. Figure shows the conditioned inhibition experiment. In this experiment CS and CS 3 were positively associated with the response i.e. they had an excitatory influence on the response (this was done during the first 7 units). After the initial acquisitions the stimuli CS and CS 2 were presented simultaneously (between 7 and 22). During the from 22 to 25 the inhibitory connection was created, since no was presented, and associated to stimulus CS 2. Finally, from 25 and forward on, the response levels of different stimuli were again tested. The activation of stimulus CS 2 has an inhibitory effect on the response. Secondary Conditioning Secondary conditioning was strong. Figure 2 shows how secondary conditioning was used to transfer the association from CS to CS 2. The transfer was made with delay A type of conditioning at to 2. The second printnow-signal at 5 was not as strong as the first at 2 because it was induced by CS. At 2 to 25 the response induced by CS 2 was tested. Print CS CS Figure 2 Secondary conditioning of CS 2 from CS. Stimulus CS 2 gave a response ( 2 to 25) that was as intense as that given from CS ( 5 to 2). Facilitated Acquisition In this experiment the effect of an intermittent stimulus during the delay period was evaluated. Figure 3 shows a trace conditioning with a ISI-period of 6. Figure 4 shows a trace conditioning but with an intermittent stimulus during the silent period

20 between 2-6. The conditioning with an intermittent stimulus during the silent period gave a slightly higher level of response (compare the levels of response at 3 in Figure 3 and Figure 4). Print CS CS Figure 3 Normal acquisition of a conditioned stimulus. The ISI-period of this trace conditioning was 6 units. Print CS CS Figure 4 Facilitated acquisition of a conditioned stimulus. The ISI-period was 6 units. At the start of the silent period, at 2, a second stimulus was activated.

21 Comparison to Other Models The results from the experiments are summarized in Table. The BCPNN-model was able to perform most of the tasks. SB TD Klopf Balkenius SD BCPNN Trace Conditioning * * * * * * Delay Conditioning * O * * * ISI-curve * O o * o S-shaped Acquisition * * * Extinction * * * * * * Reacquisition o * Blocking * * * * * * Secondary Conditioning O o * * * Conditioned Inhibition * * * * * * Facilitation * * * * * o Table This table was copied from Balkenius paper []. The last column contains data on the BCPNN-model. A * means that this feature was explained by the model and a o means that this feature could partially be explained by the model.

22 Discussion This new model has a different origin than many other models of classical conditioning. The BCPNN model was developed from the perspective: We have a biological plausible neural network model, can it be used to build a system that models the data seen in classical condoning experiments? Usually models of classical conditioning have been developed based on a desire to model the data in the experiments. An implication of this is that the modeling of the underlying neural processes has had to stand back in order for the models to meet their expected output. The approach to modeling where no attention is paid to the underlying mechanisms is taken to its full extent in the computational models. The computational models that are completely based on statistical considerations without any biological considerations represent the complete opposite to our modeling approach. The Balkenius model has large similarities to the BCPNN model. The Balkenius model uses one layer or set of differential equations to compute the weights. As in most of the models, including the BCPNN, this first set of differential equations enables the model to handle asynchronous associations. In order to handle the dynamics of the response the Balkenius model uses delay lines. What we have done in the BCPNN model is to combine these two aspects of the dynamics, the associative and response dynamics, into a single system of coupled differential equations. This means that the acquisition process has been modeled by the dynamics of synapses and membrane potentials. A hallmark of this model is the unified view that has been used when constructing the model. All three populations are built with the same neural model, the BCPNN. The system can easily be extended with more populations and incorporated with other BCPNN neural systems. In the current model the populations are only groups of units but in a more elaborate model these populations could be attractor memories or complex sensory or output systems. The BCPNN model did excellently model extinction, blocking, inhibition and secondary conditioning. The model did not fully model the ISI curve of trace conditioning. The model could not handle the situation where the was presented before the CS. In a successful model, a reversed order of stimuli presentation results in the creation of an inhibitory connection. By setting the τ Zpost to a value larger than one the connection between the CS- and R-population probably would become inhibitory. An important part of the BCPNN model is the printnow-signal. There are many ways to compute the printnow-signal and the behavior of the model is heavily dependent on it. In this paper we did not use a neural circuit to compute the derivate of the activity in the R-population. This is an area of future studies. A future study should also address the issue with reacquisition. The experiments in this report showed an opposite behavior to the desired one. The effect of repeated acquisition-extinction was that the learning became slower after the first learning trial instead of faster.

23 References. Morén, J., Emotion and Learning - A Computational Model of the Amygdala, in Cognitive Studies. 22, Lund University: Lund. p Gallistel, C.R. and J. Gibbon, Computational Versus Associative Models of Simple Conditioning. Current Directions in Psychological Science, 2. : p Johansson, C., A. Sandberg, and A. Lansner. A Neural Network with Hypercolumns. in ICANN. 22. Spain, Madrid: Springer-Verlag Berlin. 4. Lansner, A. and Ö. Ekeberg, A one-layer feedback artificial neural network with a Bayesian learning rule. Int. J. Neural Systems, 989. : p Sandberg, A., et al., A Bayesian attractor network with incremental learning. Network: Computation in Neural Systems, 22. 3(2): p Holst, A. and A. Lansner, A Higher Order Bayesian Neural Network for Classification and Diagnosis, in Applied Decision Technologies: Computational Learning and Probabilistic Reasoning, A. Gammerman, Editor. 996, John Wiley & Sons Ltd.: New York. p Sandberg, A., A. Lansner, and K.-M. Petersson. A Bayesian Connectionist Model for Memory Scanning. in XXVII International Congress of Psychology Sandberg, A. and A. Lansner, Synaptic Depression as an Intrinsic Driver of Reinstatement Dynamics in an Attractor Network. Neurocomputing, Sandberg, A., K.M. Petersson, and A. Lansner, Selective Enhancement of Recall through Plasticity Modulation in an Autoassociative Memory. Neurocomputing, : p Balkenius, C. and J. Morén, Computational Models of Classical Conditioning: A Comparative Study. 998, Lund University Cognitive Science: Lund. p... Balkenius, C. Generalization in Instrumental Learning. in From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior. 996: The MIT Press/Bradford Books. 2. Johansson, C. and A. Lansner, A Neural Reinforcement Learning System. 22, Nada, SANS: Stockholm. 3. Holst, A., The Use of a Bayesian Neural Network Model for Classification Tasks, in Dept. of Numerical Analysis and Computing Science. 997, Kungl. Tekniska Högskolan, Stockholm. 4. Hebb, D.O., The Organization of Behavior. 949, New York: John Wiley Inc. 5. Kandel, E.R., J.H. Schwartz, and T.M. Jessell, The Anatomical Organization of the CNS, Coding of Sensory Information, in Principles of Neural Science. 2, McGraw-Hill Companies. p , Marder, E. and V. Thirumalai, Cellular, synaptic and network effects of neuromodulation. Neural Networks, 22. 5(4-6): p Reynolds, J.N.J. and J.R. Wickens, Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 22. 5(4-6): p