1. Introduction. Graham Kendall School of Computer Science and IT ASAP Research Group University of Nottingham Nottingham, NG8 1BB

Transcription

1 The Co-evoluton of Tradng Strateges n A Mult-agent Based Smulated Stock Market Through the Integraton of Indvdual Learnng and Socal Learnng Graham Kendall School of Computer Scence and IT ASAP Research Group Unversty of Nottngham Nottngham, NG8 1BB gxk@cs.nott.ac.uk Yan Su School of Computer Scence and IT ASAP Research Group Unversty of Nottngham Nottngham, NG8 1BB yxs@cs.nott.ac.uk Abstract In ths paper we present a mult-agent based model of a smulated stock market wthn whch actve stock traders are modelled as heterogeneous adaptve artfcal agents. We employ the approach of ntegratng ndvdual learnng and socal learnng to co-evolve these artfcal agents wth the am of evolvng successful tradng strateges. The proposed model was tested on the Brtsh Petroleum (BP.L) share from the LSE (London Stock Exchange). Throughout the experment we see successful tradng strateges emerge among the artfcal traders. These artfcal agents also demonstrate rch dynamc learnng behavours durng the smulaton. On average, 80% of the artfcal stock traders were able to trade usng successful tradng strateges whch brngs the nvestors hgher returns compared to a baselne buyand-hold Keywords Mult-agent System, Smulated Stock Market, Tradng Strateges, Artfcal Neural Network (ANNs), Genetc Algorthm (GA), Indvdual Learnng, Socal Learnng, Co-evoluton. 1. Introducton Tradtonally the stock market has been studed usng standard representatve agent models wthout takng nto account the nature of the market where heterogeneous nvestors wth varous expectatons and dfferent levels of ratonalty nteract wth each other through the market. Palmer et al. [1] descrbed a smple mult-agent based model of a stock market nsde whch ndependent adaptve agents can buy and sell stock on a central stock market. Based on ths dea, varous types of Artfcal Stock Market (ASM) were developed [2,3,4] and they became more and more mportant n the study of the stock market see [5] for a good revew on early work on agent based computatonal fnancal markets and [6] for the recent advances n evolutonary computaton n economcs and fnance. These mult-agent based ASM models, rather than takng real data from the real world markets, buld the artfcal stock markets from the ground up usng a certan market structure together wth the artfcal stock traders modelled as heterogeneous adaptve agents. Insde these artfcal stock markets, stock prces are generated endogenously and the resultng tme seres and market dynamcs are studed [2,3,4]. Schulenberg et al. [7,8] took another approach by ntroducng real market data nto an adaptve agent based stock market model. They showed that ther artfcal agents, by dsplayng dfferent and rch behavours, are able to dscover and refne novel and successful sets of market strateges that outperform a tradtonal buy-and-hold strategy and rsk-less bond. In Schulenberg et al s model, artfcal nvestors are modelled usng Learnng Classfer Systems (LCSs). One major problem wth LCS systems s that the classfer rules are desgned explctly before the evolutonary process of the LCSs begns, thus the novelty of evolved market strateges (LCSs) s questonable. The other problem, both wth Schulenberg et al s model and other early mult-agent based ASM models, s the ambguty of the dfference between ndvdual learnng and socal learnng wthn these models. Vrend [9] dscussed the essental dfference between ndvdual and socal learnng, and ts consequences for computatonal analyss usng the experments

2 carred out n a standard Cournot olgopoly game. Vrend states that the computatonal modellng choce made between ndvdual and socal learnng algorthms should be made more carefully, snce there may be sgnfcant mplcatons for the outcomes generated. Chen et al. [4] embraced Vrend s research nto ther artfcal stock market models, and demonstrated that dfferent learnng mechansms resulted n lttle dfference n the macro-structures,.e. the econometrc propertes of the tme seres of the generated artfcal stock markets. However, dfferent learnng mechansms generated dfferent mcro-structures of the resultng artfcal stock markets regardng the traders behavour and belef. Our am here s to employ Chen et al s approach, and apply t to the real world stock market. We propose a mult-agent based smulated stock market where market scenaro, such as stock prce and tradng volume, are gven exogenously. Insde the smulated stock market, heterogeneous artfcal stock traders, modelled usng artfcal neural networks, wll trade stocks usng real market data and coevolve wth each other by the means of ndvdual and socal learnng. Our current experment, testng our model on the Brtsh Petroleum (BP.L) share from the London Stock Market, shows that, 80% of the artfcal stock traders outperformed the baselne buy-and-hold strategy and the artfcal agents demonstrate rch dynamc learnng behavours. 2. Background Chen et al. [4] dscussed the two man dfferences among the agent-based approaches for studyng fnancal markets: representaton of agents and learnng mechansm. In Schulenberg et al. s experments [7,8], three dfferent types of traders wth pre-defned types were studed. We ntend to break the constrants on these predefned traders by representng our artfcal traders usng randomly generated artfcal neural networks (ANNs). Tradtonally, artfcal stock traders modelled usng ANNs tend to use the same set of ndcators from the market whch s contradctory to the fact that dfferent people n the market receve dfferent sets of nformaton from the market. To solve ths problem, we propose a central pool of techncal ndcators from whch traders wll select ndcators to form dfferent types of tradng strateges. Ths central pool s also the mechansm through whch the socal learnng process s carred out. Ths central pool, n fact, s a smulaton of the socal culture n the smulated market. Traders are allowed to tell other traders how mportant he beleves hs ndcators are by assgnng scores them. Traders are also allowed to publsh ther successful strateges nto the central pool so that other traders can learn hs 3 The Model 3.1 Smulated Stock Market Publsh strategy Central Indcators Pool Strateges Select a new set of ndcators Fg. 1. Smulated Stock Market Copy a strategy from the pool Trader 1 50 Fgure 1 shows our mult-agent based model of a smulated stock market, whch s descrbed as follows: 1. Before tradng starts, there are 50 actve traders n the smulated stock market. There are 20 ndcators and zero tradng strateges n the central pool. The 20 avalable ndcators are assgned an equal score of 1. Each trader selects a random number of ndcators usng roulette wheel selecton. 2. Wth the set of ndcators selected, each trader generates ten dfferent models. These ten models may have dfferent network archtectures, but they use the same set of ndcators selected by the trader. The am s for the trader to evolve models from these ten by the means of ndvdual learnng. 3. The tme span of the experment covers 3750 tradng days, whch s dvded nto

3 30 ntervals. Each nterval contans 125 days (6-month tradng). 4. Each 125-day tradng s sub-dvded nto ntervals of 5 days. Each trader trades for 5 days, and then undertakes ndvdual learnng by means of a Genetc Algorthm (GA). 5. At the end of each 125-day tradng, socal learnng occurs and each trader s gven the opportunty to decde whether to look for more successful strateges from the pool or whether to publsh hs/her successful strateges nto the central pool. 6. After socal learnng has fnshed, the system enters the next 125 tradng days and steps 4, 5 and 6 are repeated. 7. For every transacton, buy means use all the cash n the trader s account and sell means sell all hs holdngs. Both margn account, where traders could buy stocks on credt, and short sellng, where traders could sell stocks she/he does not hold, and buy t back at a later tme, are not allowed. Traders are asked to pay a tradng fee of 10 for each transacton. Traders are also pad nterest for any cash n ther account, wth an annual nterest rate of 5%. Interest s calculated every half year. Except the 50 actve stock traders, there s also one nvestor usng a tradtonal buy-andhold strategy and one nvestor who saves all the money n a bank. Ther performance wll serve as benchmarks for the 50 actve traders. The buy and hold nvestor wll use all the money n the bank to buy the stock on the frst tradng day, and hold t untl the last day of tradng. The bank savngs nvestor wll sell all shares on hand on the frst tradng day, and keep all the money n a bank for the entre perod, recevng an annual nterest rate of 5%. On the frst tradng day, all traders and nvestors are gven a portfolo of,000 cash n bank and 0 BP shares. 3.2 Data and Data Pre-processng Shares of BP PLC from the London Stock Market s selected to be traded n the smulated stock market. Fg 2 shows BP s hstorcal prce. Prce (Pence) BP (BP.L) Share Prce (3/Dec/ /Jan/2003) 12/3/1987 4/17/1989 8/31/1990 1/14/1992 5/29/ /12/1994 2/25/1996 7/10/ /23/1998 4/7/2000 8/21/2001 1/4/2003 Tradng Day Fg. 2. BP PLC (BP.L) share prce Besdes the prmtve hstorcal share prce, other fnancal data s also used to compose 20 popular techncal ndcators. Ths data ncludes: tradng volume; ntra-day hgh, ntra-day low; FTSE- ndex; DJ Ol&Gas Index(UK), S&P 500 Index and DJ INDU AVERAGE. All data was acqured from Yahoo Fnancal ( Table 1 shows the 20 techncal ndcators used. Table 1. Techncal ndcators that are used as nputs nto the neural networks. All values are normalsed nto the range of [0,1]. TI Descrpton 1 10 days movng average 2 20 days movng average 3 50 days movng average days movng average 5 Closng prce (normalzed) 6 Rate of change (prce) 7 Oscllator (prce) 8 10 days bas 9 20 days volume rate of change days relatve strength days relatve strength days relatve strength 13 Stochastc oscllators (k%) 14 Fast stochastcs (D%) 15 Slow stochastcs (slow D) 16 FTSE- Index rate of change 17 Relatve strength ndex to FTSE- Index 18 S&P 500 Index rate of change 19 DJ INDU AVERAGE ndex rate of change 20 DJ Ol&Gas Index (UK) rate of change

4 4. GA and Indvdual Learnng 4.2 Indvdual learnng 4.1 Predcton Model The neural networks used by the traders are mult-layer feed-forward networks. The networks are ether 2-layer (no hdden layer) or 3-layer (one hdden layer). Two dfferent types of actvaton functon (sgmod and tanh) are used. There s one sngle output node from the network. In order to facltate the GA learnng process, the descrpton fle of each neural network s desgned n a way such that t can also be used as a chromosome wthn the GA, as shown n Fg 3. Header C1 C2 Cn Cx Fg. 3. A neural network chromosome. Each chromosome conssts of a header and a number of connectons. The header contans general nformaton about the network: startng nput node, endng nput node, startng hdden node, endng hdden node. Each connecton, Cn, contans four components: startng node (SN), endng node (EN), weght (W), and actvaton functon (AF). Durng the GA process, both the weghts of the connecton (W) and actvaton functon (AF) are mutated. Besdes the mutaton of weghts and actvaton functon, the structure of network s also evolved by means of addng a new node or deletng a node from the chromosome. SN and EN are used to keep track of the order of connectons n the neural network. As stated above, traders are allowed to use dfferent sets of ndcators for tradng. Table 2 shows the number of ndcators used by trader no. 1 to traders no. 24 on the frst day of tradng. Table 2. Number of ndcators (NOI) used by trader no. 1 to no.24 on the frst tradng day. Trader NOI Trader NOI Trader NOI Indvdual learnng occurs durng every 125-day tradng perod. At the start of each perod, each trader decdes whch set of ndcators they wll use to buld ther predcton models. Each trader bulds ten models based on ther selected ndcators. These ten models all use the same set of ndcators, but wth dfferent network archtectures. Each trader evolves hs ten models n an attempt to acheve better predcton models, usng a GA descrbed below. Durng the 125 tradng days, a model s chosen, usng roulette wheel selecton, for the next 5 days tradng. The selecton s based on the ten models scores. At the end of each 5-day tradng, trader s ROP (rate of proft) s calculated usng Formula 1. W W ROP = 10 W SN EN W AF (1) W s the trader s current assets (cash + shares). W s the trader s assets one week before. The selected model s score s then update usng Formula 2. n n m = m + ROP (2) where s trader and n s the n th model selected from the 10 models. Based on the new updated scores, four models are selected as parents, usng roulette wheel selecton. Another four models, those wth the lowest scores, are selected and wll be replaced by four new offsprng (produced by the four parents through mutaton). Overall, the four parent models selected and the two remanng models wll stay ntact and contnue to the next generaton together wth the four new offsprng. As a trader s predcton models (neural networks) has dfferent numbers of hdden nodes, possbly dfferent numbers of hdden layers and maybe uses dfferent actvaton functons, t wll not be sensble to use a crossover operator n the GA. Therefore, wthn the GA we set the probablty of crossover 0 and mutaton to 1. The complete ndvdual algorthm s gven n Fgure 4: Select models to be mutated usng roulette selecton; Select models to be elmnated; Decde number of connectons to be mutated, m; = 0; Whle( < m){ Randomly select a connecton;

5 Weght = weght + w; = + 1;} Wth 1/3 probablty add hdden node; Wth 1/3 probablty delete hdden node; replace models to be elmnated wth the new mutated models; Fg. 4. Indvdual learnng The number of connectons to be mutated, m, s a random nteger between 0 and the total number of connectons n the selected neural network. w s a random Gaussan number wth a mean of zero and standard devaton of one. Besdes the mutaton of weghts, we also evolve the structure of the network by allowng the probablty of addng or deleton of hdden nodes. After producng ten new models, the trader wll select a model for the next 5 tradng days, usng roulette wheel selecton. Indvdual learnng occurs at the end of every 5-day tradng for each trader. 5. Socal Learnng After 25 weeks (125 days) of tradng and ndvdual learnng, all traders enter a socal learnng stage. Durng socal learnng, all traders have the chance to see how other traders are performng. Traders may decde to learn from other traders, or publsh ther own successful tradng strateges. At ths stage, each trader wll carry out a self-assessment. The trader s decson n socal learnng depends on the result from ths self-assessment. Based on the methods used by Chen et al. [4], our trader s assessment s calculated usng Formula 3, 4 and 5. Frst, the traders rate of proft (ROP) (Formula 1) for the past sx months s calculated, and the 50 traders are ranked from 0 to 49 accordng to ther ROP. R S peer = 1 (3) 49 R s the rank of trader n the range of [0,49] (0 means hghest rank wth largest ROP). Formula 3 gves each trader a score n terms of peer pressure from other traders. In other words, ths score shows trader s performance compared to other traders. ROP ROP S self = (4) ROP s the rate of proft for the current sx months tradng. ROP s the rate of proft for the prevous sx months. Formula 4 gves the trader s score n terms of hs own performance n the past sx months compared to the prevous sx months. Fnally, these two types of performance are composed nto Formula 5, whch gves the overall assessment for trader. 1 assessment = S peer + (1 S ) 1 self (5) + e The fnal assessments for 50 traders are then normalsed nto the range of [0,1]. Dependng on ther assessment, a trader may choose to: 1) If a trader s assessment s 1, and the trader s not usng a strategy drawn from the pool, then publsh the strategy nto the central pool. Go nto the next sx months tradng usng the same 2) If a trader s assessment s 1, and the trader s usng a strategy coped from the pool, do not publsh t agan, but update ths strategy s score n the pool usng ther sx month ROP. Go nto next sx months tradng usng the same 3) If a trader s assessment s less than 0.9, the trader has 0.5 probablty of copyng a strategy from pool, whch means the trader wll dscard whatever model he s usng, and select a better tradng strategy from the pool usng roulette selecton, and go nto the next sx months tradng wth ths coped Or, wth 0.5 probablty, the trader wll decde to dscard whatever strategy he s usng, and select another set of ndcators as nputs, buld 10 new models and go nto next sx months tradng wth these 10 new models. 4) If assessment s between 1 and 0.9, the trader s satsfed wth hs performance n past sx months and contnues usng that Traders wll also update scores of ndcators they have used n the central pool based on ther performance n the current sx months usng Formula 6 below. n n I = I + ROP (6) where s the trader. n s the n th ndcator used by trader n the current sx month tradng. ROP s the rate of proft of the trader n the current sx months tradng.

6 6. Expermental Result The man consderaton of choosng the BP share as the test bed for our model s that hs share s a upturn stock n the overall trend, see fgure 2. A buy-and-hold strategy wll, obvously, brng the nvestor a postve return. If our artfcal traders could acheve a hgher return than the classcal buy-and-hold strategy through both ther ndvdual and socal learnng, that means these artfcal agents have dscovered successful tradng strateges durng the evoluton. The results from our experment proved successful tradng strateges have been developed. Wealth (Sterlng Pounds) Bank Savngs Buy and Hold x Tradng Day Fg. 5. Traders Performance (BP.L share) Fgure 5 demonstrates the growth of wealth of our 50 artfcal traders durng the 3750 days tradng perod. The thck black lne ndcates the growth of wealth of the nvestor wth buy-andhold The thn lne at the bottom of the dagram wth o markers ndcates the growth of wealth of the nvestor who saved hs money n bank for 15 years. From Fgure 5 we can see the majorty of the artfcal traders (represented by the lnes above the thck black lne) were able to learn to predct the trend n the stock,.e. start to buy n stocks when the share s gong up and start to sell t when t s gong down. The more accurately the trader was able to control the tmng of buyng and sellng, the faster they accumulated wealth. Table 3 gves the statstc results on the 50 artfcal traders. It shows 40 out of the 50 artfcal traders beat the buy and hold On average, 50 actve traders outperform buy and hold strategy by 25.84%. Table 3. Results from 50 traders compared wth buy and hold strategy and bank savngs. All returns are calculated as the dfference between wealth on day 3750 and the nvestor s orgnal wealth dvded by the orgnal wealth. Descrpton Result Return from bank savngs % Return from buy-and-hold strategy % Average return from the 50 traders % Maxmum return among the 50 traders % Mnmum return among the 50 traders 57.7% No. of traders who outperform savngs 49 No. of traders who outperform buy and 40 hold To see the rch learnng dynamcs of 50 artfcal agents more clearly, we selected 3 traders from the 50 traders and depct them n Fgure 6. Trader 14 s the best performer who acheved a return of % on hs orgnal wealth. Overall, trader 14 was able to predct the trend n the prce of the stock farly well. The transacton records of trader 14 shows ths trader, n fact, learned a strategy from the central pool n the early days, and kept t untl the last tradng day. Some other traders also used the same strategy coped from the pool, but trader 14 refned ths strategy constantly through hs own ndvdual learnng whch results n hs outstandng performance compared to others. Trader 16 s the worst performer whose wealth lne fnally runs below bank savng lne. Ths trader, showed by hs transacton records, dd not consult the central pool for other trader s strategy throughout the whole tradng perod. He bascally followed the buy-and-hold strategy n the early stage of the tradng perod. The trader s own ndvdual learnng dd not help hm too much. Around day 2200, he made a mstake by sellng the stock when the prce was stll gong up. When the stock prce dropped dramatcally around day 3000, ths trader s strategy completely faled. On the contrary, trader 2, shown by hs transacton records, constantly searched for good strateges from the pool, and tred out dfferent strateges durng the dfferent stage of the tradng perod. Before day 2300, trader 2 used a strategy learned from central pool whch worked qute well durng the upturn perod. However, durng the downturn after day 2300, ths strategy ddn t work very well. Trader 2 went on and tred dfferent strateges from the pool, fnally stll managed to outperform buyand-hold

7 In summary, our artfcal agents demonstrated rch dynamc and nterestng learnng behavours durng the smulaton whch s very smlar to real stock traders n the real world market. The mechansm of ntegratng ndvdual and socal learnng here played an mportant role n the sense of agents learnng behavours and abltes. Wealth (Sterlng Pounds) Trader 2 Trader 14 x 10 3 Trader 16 Bank Savngs Buy and Hold Tradng Day Fg. 6. Traders Learnng Dynamcs 7. Concluson Compared to the traders wth specfcally predefned types studed n Schulenberg et al. [7,8], our 50 artfcal traders were generated completely randomly wthout defnng what types of ratonalty or belef they should have. These 50 artfcal stock traders, mtatng real world traders, traded a stock n a smulated stock market, learned to trade by themselves and learned from other traders through socal learnng. The results from the smulaton shows 80% (40/50) of our artfcal agents learned successfully n tradng stock, and outperformed the baselne buy-and-hold Moreover, these 50 randomly generated artfcal traders demonstrated more dynamc and nterestng learnng behavours durng the smulaton compared wth the three dfferent types of traders studed n Schulenberg et al. s experments. It wll be very nterestng to see how dfferent learnng mechansms, for example, a solo ndvdual learnng process, a solo socal learnng process, compared to a ntegrated ndvdual and socal learnng process, affect the artfcal agents learnng behavours and abltes. Ths s one the drecton of our future research. References [1] Palmer, R.G., Arthur, W.B., Holland, J.H., LeBaron, B., Tayler, P.: Artfcal economc lfe: a smple model of a stock market. Physca D, Vol. 75. (1994) [2] Arthur, W.B., Holland, J.H., Lebaron, B., Palmer, R., Tayler, P.: Asset Prcng under Endogenous Expectatons n an Artfcal Stock Market. The Economy as an Evolvng Complex System, XXVII. Addson-Wesley (1997) [3] Lebaron, B., Arthur, W.B., Palmer, R.: Tme Seres Propertes of an Artfcal Stock Market Model. Journal of Economc Dynamcs and Control (1999), [4] Chen, S.H., Yeh, C.H.: Toward an ntegraton of socal learnng and ndvdual learnng n agent-based computatonal stock markets: The approach based on populaton genetc programmng. Journal of Management and Economcs, 5, (2001) [5] Lebaron, B.: Agent based computatonal fnance: Suggested readngs and early research. Workng paper, Graduate School of Internatonal Economcs and Fnance, Brandes Unversty, October (1998) [6] Chen, S.H.: Evolutonary Computaton n Economcs and Fnance, Physca-Verlag, (2002) [7] Schulenberg, S., Ross, P.: An Adaptve Agent Based Economc Model. In: Lanz, P.L., Stolzmann, W., Wlson, S.W. (eds): Learnng Classfer Systems: From Foundatons to Applcatons, Vol of LNAI. Sprnger-Verlag, (2000) [8] Schulenburg, S., Ross, P.: Strength and Money: An LCS Approach to Increasng Returns. In Lanz, P.L., Stolzmann, W., Wlson, S.W., (edtors). Advances n Learnng Classfer Systems, volume 1996 of Lecture Notes n Artfcal Intellgence, Sprnger-Verlag, Berln, (2001), [9] Vrend, N.: An llustraton of the essental dfference between ndvdual and socal learnng, and ts consequences for computatonal analyss. Techncal report. Queen Mary and Westfeld College, Unversty of London (1998)