Automated Stock Trading and Portfolio Optimization Using XCS Trader and Technical Analysis. Anil Chauhan

Transcription

1 Automated Stock Trading and Portfolio Optimization Using XCS Trader and Technical Analysis Anil Chauhan Master of Science Artificial Intelligence School of Informatics University of Edinburgh 2008

2 Abstract Financial market is highly dynamic system for which finding underlying price pattern is highly complex. We have extended the previous work done on automatic stock trading using extended classifier system (XCS) by implementing Q (1) and Q (λ) Reinforcement Learning algorithm. We developed 14 XCS agents using different technical indicators like Moving averages,rsi,cmf,sar,adx etc. We showed that by modeling financial prediction as single step reinforcement learning problem and using the concept of delayed reward for checking correctness of action taken, all the benchmarks strategies like buy and hold, 'keeping money in bank' etc could be beaten. We have also shown that stock price movement is co-related with other day price movement and reformulated the financial forecasting as a multi step process. We introduced the concept of passive set and found that multi step problem formulation gives best results. Q learning gave 18% better performance than single step reward only RL. Finally we build a portfolio management and optimization system which learns online and does monthly or quarterly rebalancing using the best trader to trade. The results showed that reacting to the market dynamics doesn t necessarily give us the best result. We showed that such a system give us average performance between the best trader and the worst trader. We also employed different trading strategies like using more than 1 best agent and mean reversal strategy to do portfolio optimization. ii

3 Acknowledgements I would like to thank my supervisor Sonia Schulenburg for introducing me to the world of Finance and Classifier Systems and for giving constant feed back on my project. I would also like to thank Abu ul Hassan for sharing previous version of XCS java code with me. Many thanks to my friend Santosh for reviewing my initial draft of thesis and sharing ideas on the same. iii

4 Declaration I declare that this thesis was composed by me, that the work contained herein is my own except where explicitly stated otherwise in the text, and that thesis work has not been submitted for any other degree or professional qualification except as specified. (Anil Chauhan s @sms.ed.ac.uk) iv

5 Table of Contents 1 Introduction Introduction and Purpose Motivation Objective Outline Background & Related Work Background Market Efficiency Version of Efficient Market Hypothesis (EMH) Technical Analysis The Portfolio: Why do we need Portfolio? Portfolio Management: Related Work Machine Learning in Finance and Portfolio Management XCS Introduction from Stock Trading Perspective XCS Input and Output XCS Frame Work [15] XCS Learning Cycle Updating XCS Parameters Genetic Algorithm role and rule evolution[15] Deviation from other LCS based Systems Mind of XCS System Implementation Technical Analysis Usage in XCS Description of individual technical Indicators Combining different technical indicators and working mechanism of different Agents...24 v

6 Composition of 14 Agents: Advantage and Scope of Improvement of current approach Improving the learning of extended Classifier System Classifiers in multi step Reinforcement learning problems Implementing Q learning in Classifier Eligibility trace and Watkins's Q(λ) Experimentation FTSE data and Stability of the XCS System FTSE Data Stability of the XCS System Comparative Study of the 3 different Algorithm Setting the parameters for the experiments Setting Initial Exploration Rate Setting discount rate (gamma) Setting Trace Decay Parameter Experimental Results for 3 learning Algorithm Observations: Fault in the previous Reward giving strategy Experiments with improved delayed reward Strategy Setting Initial Exploration Rate Setting Discount Rate (gamma) Setting Trace Decay (λ) Results with new delayed reward strategy Observations: Experimental Results for all 3 learning algorithm with new delayed reward strategy Observations Fault in Q (1) learning Experimentation with passive Set Finding optimum parameters Observations for Passive set...55 vi

7 5 Implementation & Experimentation-Portfolio Optimization Implementation Portfolio Performance: Results...61 Portfolio Management Results Observations: Portfolio Management results Analysis and Comments on Portfolio Management System Change of Portfolio construction Strategy Results of Portfolio Management using more than 1 Agent Experimentation with Portfolio Management taking few best companies in the Portfolio Steps Followed Results Observation: Experimentation with Mean Reversal Strategy Results Observations Conclusion & Future Work Conclusion Future Work...71 Bibliography...72 Appendix...76 vii

8 List of Figures Figure1 XCS Frame Work...13 Figure2: Voting Strategy...17 Figure3: Q-learning: An off-policy TD control algorithm Figure4:Back ward view of eligibility trace...30 Figure5: Tabular version of Watkins's Q (λ) algorithm...31 Figure6: Mean Performance of 5 companies for different number of runs...34 Figure7: DSGI Price Chart...46 Figure8: DSGI, Wealth Chart of agents with old reward strategy...46 Figure9: DSGI, Agent s performance with new delayed reward giving strategy...47 Figure10: LAND, Price Chart...47 Figure11: LAND, Meta Agents wealth chart with old reward strategy...48 Figure12: LAND, Meta Agents wealth chart with new delayed reward strategy...48 Figure13: Portfolio Management Results...62 Figure14: Portfolio management using either best 5 or best 10 or best 20 companies...66 Figure15: Portfolio Management System using trend reversal strategy...67 Figure16: Comparison mean reversal strategy with normal strategy...68 viii

9 List of Tables Table1. Composition of different Agents...26 Table2: Details of 10, FTSE 100 Companies...32 Table3: Experimental results for 100 run on 5, FTSE100 companies...33 Table4: Setting Exploration rate...35 Table5: Combined Results for Different exploration rate...36 Table6: Setting discount rate...36 Table7: Setting trace decay parameter...37 Table8: Experimental Results for 3 learning methodology...39 Table9: Setting Exploration Rate...41 Table10: Setting discount rate...42 Table11: Setting trace decay...42 Table12: Comparative results for 90 FTSE 100 companies with delayed reward strategy for single step reward only RL...45 Table13: Results of 90 FTSE100 Companies for different RL with new delayed reward strategy...50 Table 14: Comparative Results for FTSE 100 companies using Passive Set...54 Table15: Comparison with Only active set and with additional passive set approach...55 Table16: Combined Results single step RL without Passive set and multi step Q with passive set...57 Table17: Portfolio Optimization using single best Agent...61 Table18: Monthly Portfolio management Using Best 3 agents...64 Table19: Monthly Portfolio re-balancing using different Number of Best Agents...65 Table20: Portfolio Management System with quarterly re-balancing taking best 5 companies...65

10

11 Chapter 1 1 Introduction 1.1 Introduction and Purpose Advances in modern machine learning such as evolutionary computation have enabled us not only to analyze data more efficiently, but also to understand any underlying patterns present in the financial market. This effective exploitation of new computation methods will help organizations to make better informed decisions which will further improve their competitive edge [5]. Many different approaches like Neural Networks (NN), Genetic Algorithms (GA) have been widely applied to predict the financial market. However, for some of these systems the input data might not be as rich in information content as technical indicators such as various types of moving averages, break out rules, maximum and minimum prices in the preceding days or fundamental indicators such as dividends, interest rates and money supply. More recently, the academic world has shown some promise in the area of learning classifier systems (rule based models) by Overcoming some of the most common drawbacks neural models present to practitioners, such as the lack of explanatory power, high variance in results and the need to continually retrain the nets when performance starts to decrease. In addition, very few models have addressed the integration of more than one learning paradigm within a single platform. In this project, first we will try to improve the learning of the classifier system by incorporating different Reinforcement learning algorithm. In all the previous version of extended Classifier Systems (XCS), financial forecasting was solely considered to be a single step Reinforcement Learning problem. However, we believe price movement is not completely erratic and prices do follows patterns of ups and downs. Price of any day affects the coming day prices and is in some way co-related. Due to this we will try to model financial forecasting as a multi step process and 1

12 implement Q (1) and Q (λ) Reinforcement Learning algorithm. Secondly, we will investigate the methods of portfolio construction and portfolio optimization. This will be achieved by using a system which evolves technical trading agents, each learning to trade stocks by modeling groups of traders using a variety of sets of technical indicators. For portfolio construction, the main task will be to build a portfolio management system attached to XCS which picks the best agents that can give the maximum benefit in the long run. For simplicity, the equity market will be the prime focus for this investigation, although (if time permits), tests could be extended to foreign exchange to address scalability of the approach. 1.2 Motivation The financial market is a highly dynamic system which depends on multiple factors such as bank interest rates, company base strength and everyday changing news. The motivation behind this project is to develop learning systems that can learn in an online fashion and cope with rapidly changing financial market environment. The main idea is to develop set of robust agents which uses Technical Analysis information and different Reinforcement Learning Algorithm to do automatic stock trading using extended Classifier System (XCS). They should be robust in the sense that they should be able to trade profitably (beat different benchmark strategies) on all FTSE 100 stocks. Finally we wish to build a portfolio optimization system which will interact with this modified extended Classifier System (XCS) and harnesses the strength of best agents and use different strategies like taking best companies in the portfolio, mean reversal strategies etc to do monthly or quarterly portfolio rebalancing. 2

13 1.3 Objective The project has been subdivided into following sub tasks -: 1) Optimize the problem formulation for XCS: The agents present in current system converts the information given by the technical indicators into Input (Binary String) for the XCS. The XCS System further tries to learn the optimum decision it should take when faced with a particular combination of binary bits. The better we define the problem (i.e. combine the technical indicator information) for the XCS, the better it will learn the underlying price pattern and more robust and reliable decisions (buy, sell or hold) we expect it to give. The aim is to develop such robust agents. 2) Formulate automatic stock trading and financial forecasting as a multi step problem instead of single step problem by implementing Q learning and Q (λ) learning algorithm. 3) Experiment with reward mechanism of XCS reinforcement learning portion to see if the delayed reward feedback is better than currently employed immediate reward feedback. 4) Explore the possibility of either giving negative reward to the agents for taking any incorrect action or creating and rewarding passive set for the incorrect decision taken by an agent. 5) Build a Portfolio Management System attached to the current XCS System. The portfolio management system will be responsible for -: a) Portfolio construction using different trading strategies like "utilizing combinations of best agents instead of a single agent", "mean reversal strategies" etc. b) Optimize the portfolio by pro-active monthly, quarterly or yearly rebalancing. 6) Compare the performance of the trading agents against benchmark agents like buy and hold, bank etc. 3

14 1.4 Outline The rest of this thesis is organized as follows: Chapter 2 gives the background information on the subject and briefs the readers about the current state of the art in Financial forecasting domain. Chapter 3 talks about how we have implemented different agents and learning algorithm as proposed in this thesis. It also talks about what errors we found during implementation and how we changed our approach to tackle different problems. Chapter 4 presents the experimental results obtain after implementing different Learning algorithm and our critical analysis of the results. Chapter 5 presents the implementation and experimental results for portfolio construction and optimization system and critical analysis of the same. 4

15 Chapter 2 2 Background & Related Work 2.1 Background Market Efficiency Maurice Kendall in his random walk experiment (1953) [16] found that the stock prices are completely random and has no relation to the past performance. The unpredictable price movement seems to confirm the irrationality of the market. However, on deeper analysis it became apparent that random price movement indicates a well functioning or efficient market and not an irrational one [17]. In its most basic form Efficient Market Hypothesis says that markets are information efficient i.e. all the available information that could be used for profit making quickly gets absorbed in the stock prices and the prices may increase or decrease only in response to new unavailable and unpredictable information Version of Efficient Market Hypothesis (EMH) There are 3 forms of EMH, which differs in what all available information is composed of. 1) Weak Form hypothesis states that stock prices already reflect all the market trading information like past price, volume movement etc. It means that if any form of past price or volume data movement could be used be generate reliable trading signal then all investors would have used them by now making the information fruitless[17]. It suggests that any form of Technical analysis is useless. 2) Semi strong form hypothesis states that any publicly available information like prospects of firm including fundamental data on the firm s product line, quality of management, balance sheet composition, and patents held, earning 5

16 forecast and accounting practices must also be already reflected in the stock prices [17]. This hypothesis makes the fundamental analysis also useless. 3) Strong form hypothesis states that stock prices also reflect information available only to company insiders. Such information also generally gets spread very quickly, leaving very less room for making profits. Summarizing, we can say that if there is any pattern or information that is exploitable, then mass of astute investors would attempt to profit from such predictability, which would ultimately move stock price and cause the trading strategy to self destruct Technical Analysis Technical analysis is mainly the search for recurrent and predictable patterns in the stock prices by using the past price or volume data. Technical analysis like weather forecasting doesn t result in absolute prediction about the future but help investors anticipate what is most likely to happen to the prices over time. Dow Theory lies at the root of technical analysis. 2 important points from Dow Theory are -: 1) Prices discount everything. Current price of stock fully reflects all the information. Technical analysis utilizes the information captured by the price to interpret what the market is saying with the purpose of forming a view on the future [18]. 2) Price Movements are not totally random. Most technicians believe that there are inter spread period of trending prices in between random fluctuations. Technician aim is to identify the trend and then make use of it to trade or invest. More detail about technical analysis and how we have used it in our XCS System is presented in section The Portfolio: A portfolio is a combination of different investment assets mixed and matched for the purpose of achieving an investor's goal(s). A portfolio can be viewed as a piechart where each portion represents an allocation of the investment [6]. 6

17 Why do we need Portfolio? The aim of portfolios is diversification. Different securities perform differently at any given point in time, so the idea is that with a mix of assets, the entire portfolio would not suffer the impact of a decline of any one security. It s like following the simple practice of not putting all your eggs in one basket. Spreading investment across various types of assets and markets reduces the risk of catastrophic financial losses Portfolio Management: Portfolio management is defined as the art and science of making decisions about investment mix and policy, matching investments to objectives, asset allocation for individuals and institutions, and balancing risk against performance [6]. It is an attempt to maximize return at a given appetite for risk. In the case of mutual and exchange traded funds (ETFs), there are two forms of portfolio management: passive and active. Passive management simply tracks a market index, commonly referred to as indexing or index investing. Active management usually involves a single manager, co managers, or a team of managers who attempt to beat the market return by actively managing a fund's portfolio through investment decisions based on research and decisions on individual holdings. Closed end funds are generally actively managed. 2.2 Related Work Fama s Efficient Market Hypothesis [20] (Section 2.1.1) and the Martingale Model [21], [22] rules out any strategy or publicly available information or private or return/dividends information, or use of technical analysis for excessive market returns. There are proponents on both sides who believe we can somehow predict the price and others who believe the prices are completely random. For example Burton J. Malkiel in his Random Walk experiment [23] [24] showed that prices are completely random, whereas MIT's Prof. Andrew Lo and Craig A. MacKinlay [25] published work, points out that there is a long trend in the prices. Lo and MacKinlay investigated the weekly US stock from 1962 to 1985 and found that random walk hypothesis could be easily rejected.they also described several techniques for detecting predictabilities and evaluating their statistical and economic significance. Work done by Pin Chen and Mu 7

18 Yen Chen [7] using an XCS based decision support system with technical indicators has shown promises to predict stock price fluctuations efficiently and generated good returns. Schulenburg [10] in her PhD research developed an LCS model of artificial traders and tested it in the stock market using several groups of technical indicators. Stone [26] in his PhD work applied ZCS on foreign exchange market. Competitive returns were generated in their work in most of the cases, which suggests LCS models can be successfully applied when modeling financial markets. More recently, Chen, Lin [8] used XCS for predicting future market price movements. The model used moving averages of price and volume for constructing environmental message. Gershoff and Schulenburg [9] explored the collective behavior of XCS agents to achieve accuracy in prediction Machine Learning in Finance and Portfolio Management Moody and Saffell [1] presented methods for optimizing portfolio by using an adaptive algorithm, Recurrent Reinforcement Learning (RRL), for discovering investment policies. They demonstrated how direct reinforcement can be used to optimize risk adjusted investment returns (including the differential Sharpe ratio) while accounting for the effects of transaction costs. The RRL algorithm learns profitable trading strategies in two ways: Maximize risk adjusted return as measured by Sharpe ratio. They used a modified derived form of Sharpe ratio called differential Sharpe ratio for online optimization of trading system. Avoid the downside risk by maximizing the Downside Deviation (DD) ratio, which is defined as square root of the average of the square of the negative returns. Using DD as measure of risk they used downside deviation ratio DDR to measure the utility function. RRL trader performed far better than Q trader and enables a simpler problem representation, avoids Bellman s curse of dimensionality and offers compelling advantage in efficiency. GAO and Chan [2] presented a trading and portfolio management system called QSR which uses Q learning and Sharpe ratio algorithm. They used absolute 8

19 profit and relative risk adjusted profit as performance function to train the system respectively. The experiments conducted on trading example based on foreign exchange rate showed promising results. Neuneir [3] formalized asset allocation as a Markovian Decision Problem and optimized it using dynamic programming and Q learning algorithms. Neural networks were used for value function approximations. Experimental results on German Stock market showed this strategy to be better than heuristic benchmark policy. Schulenburg and Ross [29][30] developed a LCS model where trader used technical indicators to predict the price of IBM stocks. The system was able to beat all the benchmark agents. Kyong and Sungky [4] used genetic algorithms to propose a portfolio optimization scheme for index fund management. Index funds are designed to copy the benchmark index with relatively small number of stock. The paper reported that index fund could improve its performance greatly with the proposed GA portfolio scheme. There proposed scheme is based on three fundamental variables: Portfolio beta, trading amount and market capitalization. They demonstrated the results, for index fund designed to track the Korean Stock Price Index. Dempster and Jones [11] aimed to develop an adaptive trading system that trades profitably by emulating the behavior of technical traders who adapt to the market by changing it trading strategies. There trading system uses Genetic programming to find the best combination of technical indicator to trade. The genetic algorithm can chose the combination of technical indicator from initial set of 6 technical indicators namely AMA, CCI, MACD, MA Crossover, Price Channel, RSI and Stochastic. They used a modified form of Sterling ration to gauge the performance of trading strategies. S = Return/ (1 + modified drawdown) Alongside finding trading strategies via genetic programming they also tried to optimize the built portfolio by quarterly re optimizing it. There experimental results showed that such a system which uses combination of technical indicator can make profit. The best strategy employed was able to give a return of 7% pa. However on an average they weren t able to beat buy and hold strategy. They also showed that trading in adaptive manner wherein quarterly optimization of the trading strategy is done is 9

20 ultimately loss making, which highlight the penalty for over-reaction to short term market behavior [11]. Schulenburg and Wong [12] experimented on Portfolio allocation using XCS System by combining input data using technical analysis, general market condition and options market conditions. There best performing agents performed substantially better than benchmark agents like buy and hold, trend following, bank agent and random agent. However, XCS agent s performance varies depending on initial random seed chosen and a single best performing agent can t tell much about the performance of overall system in general. Dempster, Payne, Romahi and Thompson [13] used Technical indicators for Intraday FX Trading using Reinforcement Learning and Genetic programming technique. The set of technical indicator used by them were price channel break out, adaptive moving average, relative strength index, stochastic, moving average convergence divergence, moving average crossover, momentum oscillator and commodity channel index. The performance of the System was judged on the basis of Sharpe ratio and sterling ratio. There experiments were able to generate significant insample and out of sample profits. However none of the methods produces significant profits at realistic transaction costs. 2.3 XCS Introduction from Stock Trading Perspective XCS stands for extended Classifier System. It is an accuracy based classifier system which is different from other classifier in the way that classifier fitness is derived from estimated accuracy of reward predictions instead of from reward prediction themselves. It is an Online learning machine, which improves its behavior with time through interaction with environment. XCS learns through reinforcement and the aim is not only to get more reward but to maximize the value (Summation of all the rewards in long run). The System is given least amount of prior information, so that most of the machine knowledge results from adaptation to the environment. We don t tell it how to do things but let it learn through fed inputs, action it takes and the reinforcement it gets. If it does well, we give it positive reward else penalize it in some form. 10

21 2.3.1 XCS Input and Output XCS Input Unit : The input to XCS is binary vector e.g where each bit can be thought of as crossing the threshold of continuous valued Output of some sensors. In our XCS, everyday the system gets previous day data about the individual stock price (open, high, low and close) and volume. The Meta agents present in the system apply technical analysis on this information to get buy (1), sell (0) signals. A very simple example is moving average of price and volume. Let s say a particular agent calculates the moving average of price and volume for the past 10 days. If the closing price of stock is greater than 10 day moving average of price, it suggests price may go up and that s a buy (1) sign. Similarly, if the volume of the stock is greater than 10 day moving average of volume, it also predicts a buy (1) sign. So the input string that will be fed into the XCS will be 11. This is a very simple example. In actual, system uses more advance technical analysis to form the input binary string which may range from 6 to 9 bits in length. Each bit can be either 0 (sell) or 1 (buy). More detail about how the problem is being defined to the XCS System using technical analysis can be found in section 3.1. XCS Output: XCS output is discrete action or decisions. For example in our case it is either 0(sell) or 1(buy). The final aim of learning cycle for XCS is to learn what action it should take for a particular combination of input binary bits. Please note XCS uses unsupervised learning (Reinforcement Learning) wherein at any point we don t tell it what is right and what is wrong. It has to find this out through experimental trial and error and reward mechanism. Depending on the input string sometimes it is easy to predict what the correct action is. For example if input string is i.e. out of 6 bits, 5 bits are suggesting to buy the stock, then the correct action must be to buy the stock. However, at other times the correct action might not be so evident. For example if the input string is , the proportion for both buy and sell signal are equal and we expect it to learn what weight would be appropriate to give to individual signal. Even in real life scenario, on a given day an actual trader might face with situation wherein there is no clear sign of buy or sell from combination of technical indicator. He/She then have to judge from experience which technical indicator information should be given more weight and decide accordingly. 11

22 2.3.2 XCS Frame Work [15] XCS contains population of Classifiers. Each classifier in the population is characterized by 5 main components -: 1) Condition part C, which specifies on what problem instances the classifier is applicable. 2) Action part A, specifies what action classifier takes when condition C is fulfilled. 3) Reward prediction P, estimates what payoff or reward classifier can expect on executing the action. 4) Reward prediction error ε estimated the mean absolute difference of R with respect to the actual reward. 5) Fitness F estimates the scaled, relative accuracy of classifier with respect to other overlapping classifiers in the action set it is present. In Short a classifier is a set of <Condition> : <Action> => <Prediction> Prediction is similar to payoff of Reinforcement Learning. Eg 01#1## : 1 => {0 sell : 1 Buy : # : don t care} This classifier says if first bit is 0, second is 1 and fourth is 1 and I don t care about others then after taking action 1(Buy), will be the payoffs. The payoffs are updated with the learning of system. The above given classifier s condition matches with following 8 input string. It might be the sum of all there payoffs There can be 8 such cases It s different from other action based systems like Neural Network in the sense that in Neural Networks payoff information for any Input are distributed over the whole Network. Each classifier acts only a subset of problems. It checks whether given condition is one on which it can act. If condition is there, it acts on it and predict certain payoff. 12

23 Figure1 XCS Frame Work [15] [P]: Classifier population F: fitness of prediction α 1\ε [M] : Match Set p: predicted value ε: error in prediction XCS Learning Cycle In the starting the classifier population [P] is generally empty. The agents use the stock price and volume data to form the input string (For details please see section 3.1). The Input is fed into classifier population [P], which detects if there is any match. The 4 classifier marked with -- in fig 1 matches the Input They are put in a match set [M]. If no classifier matches the given input, XCS creates classifier by covering mechanism (A rule is created at random and has random action and is assigned a low prediction). A new rule has a certain number of don t care sign (#) in random position. The # sign give classifier an initial generality due to which it can be tested on many input problem instances. 13

24 Covering is necessary only initially and vast majority of new rules are derived from existing rules. For example, suppose Input string is and there is no classifier which matches this input.then the rule created is 1##0010# : Continuing the process, after creation of Match set, XCS estimates payoff for each possible action by forming a prediction array P (A). In fig1, 2 classifiers in the match sets are predicting 01 and two are predicting 11. We take the Fitness weighted average of prediction for each action Predicted weighted = Σ prediction * fitness Average Σ fitness Eg P(action=01) = 43* * = Similarly P(action=11) = 16.6 Hence, P(A) shows fitness weighted average of all reward prediction estimates of the classifier in [M] that advocate classification A. The System follows an ε greedy policy i.e. it takes the best action most of the time, but with small probability ε (exploration probability) it also takes suboptimal action and chooses random action from those in the prediction array. All classifiers in match set [M] that specifies chosen action A forms the action set [A]. In fig 1, we have chosen the action with maximum prediction i.e. 01 and 2 classifiers having this action are put in action set. The System executes the prescribed action. Next day the correctness of the action taken is judged by the stock price movement. For every correct action a reward of 1000 is given. For wrong action reward of 0 is given. It differs from normal Reinforcement Learning methodology in the sense that for incorrect action, negative reward is not given. For example let s suppose the system predicts rise in the price of stock and it buys the share. If next day the prices go up, then a reward of 1000 is given. This reward is used to update the parameters of classifiers in action set [A]. 14

25 Updating XCS Parameters Initially on creation of a classifier, it is given a very low prediction value. After getting the reward for the executed action, its parameters are updated as follows-: Prediction α Is learning rate (~ 0.2) : Pj Pj + α(r - Pj) so if R > Pj then Pj value is increased i.e. it s prediction will go up. As can be seen, if this particular classifier is updated many times, Pj will tend toward R i.e. predicted value will tend towards the actual return from the process. Similarly, other parameters are updated as Error Accuracy : Ej Ej + α( R Pj - Ej) : Kj == Ej m if Ej > Eo else Eo n Relative : Kj = Kj / Σ Kj over [A] Accuracy Relative accuracy shows relative accuracy of classifier with respect to classifiers in action set. Fitness : Fj Fj + α(kj - Fj) Fitness of the classifier is an estimate of its accuracy with respect to accuracies of other classifiers in the action set it occurs Genetic Algorithm role and rule evolution[15] XCS applies Genetic algorithm for rule evolution. If the average time since the last GA was applied, exceeds certain threshold then genetic reproduction is invoked in the current action set [A]. The GA selects 2 parental classifier based on there relative fitness in action set [A]. Two offspring s are generated reproducing the parents by applying crossover and mutation. Parents and offspring s both compete in the same population [P]. Niche mutation is applied in the classifier which means that the mutated classifiers still matches the current problem instance or input binary string they were able to act previously. If the offspring condition is subsumed by some other classifier than it is not inserted into the population and only the numerosity of the subsumer classifier is increased by 1. The classifier population is fixed and deletion is 15

26 done if over populated. Excess classifiers are deleted from [P] with probability proportional to the action set size estimate that the classifiers occur in. If classifiers are more experienced with less fitness there probability of deletion is more [15]. For more information on this, readers are encouraged to read chapter 4 of martin butz book. The classifiers which are more general will more often be part of an action set and thus undergo more reproduction events and thus propagates faster. Thus the GA process is expected to evolve the accurate, maximally general solution as the final outcome. For example the below mentioned classifier undergoes cross over to give offspring on the right hand side. 1 0 # # 1 1 : # # 1 # : (1) # # : 2 # : (2) Please note result of crossing are : A classifier (1) which is more general than both, A classifier (2), which is more specific than both. A more specific classifier can never be less accurate. It is not the case always but the process tends on balance to search along generality specific dimensions, using piece of existing higher accuracy classifiers. It is clear that population will tend towards having classifiers with greater accuracy [15] Deviation from other LCS based Systems XCS reproduces classifiers selecting from the current action set instead of from the whole population. Relative accuracy based fitness measure the performance of a classifier. Reproduction favors those who s condition matches and come more often in the action set. Deletion occurs from whole of the population. 16

27 2.3.5 Mind of XCS System Figure 2: Voting Strategy [19] Our modeled XCS System in its basic form consists of 7 Agents which uses different set of technical analysis information to create the Input binary string. Each agent has 25 copies which simultaneously do the trading and prediction. One voting agent combines the prediction of these 25 agents and presents it to the meta-agent. The system learns in an online fashion. There are two separate phases, learning phase and trading phase. During the learning phase all the agents simply explores and updates the parameters of classifiers and no actual money is invested. During the trading phase, out of 25 agents, system randomly picks some agents who explores (take random sub-optimal action) and other agents exploit (take best possible action which is supposed to give maximum reward). While combining the decision, voting-agent consider the factor of current wealth of 25 agents and discard the decision of those agents who are loss making. Also any agent who is exploring (taking random action), his action is not taken into account. Meta agent finally takes the decision of either to buy or sell using the composite predictive power of 25 XCS Agent. For portfolio management system 14 different types of XCS Agents were used. Due to continuous process of exploring and exploitation even during the trading phase, learning of the system never stops. Due to this continuous learning, if the dynamics of the market changes, we expect system to capture those variations also. 17

28 Chapter 3 3 Implementation 3.1 Technical Analysis Usage in XCS Technical analysis overall is more of an art than a science. There is no single kind of technical indicator which can work for all the stocks in the market. In our XCS System technical analysis information is used to make the input binary string. For our purpose we used and coded 14 individual technical indicators. There are open source library for technical indicators. However, we have coded our own set of technical indicators, so that some form of heuristic can further be applied to individual technical indicators to generate more robust buy or sell signal. For example one such technical indicator, Relative Strength Index (RSI) ranges from 0 to 100 and gives over bought and over sold condition for RSI greater than 70 and RSI less than 30 respectively. We used heuristic to generate buy or sell signal in the range 30 and 70.More details about this can be found in Section This section is further divided into following parts -: Description of individual technical indicators and how they are used to generate buy or sell signal Description of agents which combines different technical indicator information. 3. Advantages and scope of improvement of the current approach. 18

29 3.1.1 Description of individual technical Indicators Technical indicator is defined as a series of data points that are derived by applying a formula to the price data of security which can be combination of the open, high, low or close over a period of time [18]. These data points can be used to generate buy or sell signal which we shall shortly see. Technical indicators can provide unique viewpoint on the strength and direction of the underlying price action. Different technical indicators employed in our XCS System are -: Moving average: It is a lagging indicator which simply calculates average price of security over a specified number of periods. Moving average filters out random noise and offers a smooth perspective of price action. They work well when stock develops a strong trend. Usage of Moving average: In our XCS System Moving average is used in 2 ways to generate buy and sell signal. The location of current price, relative to the moving average: 10 and 20 day moving average is used for this purpose. if (MA10[index]-close>=0){binary+="0";}else{binary+="1";} if (MA20[index]-close>=0){binary+="0";}else{binary+="1";} Location of shorter moving average relative to longer moving average. if(ma20[index]>ma10[index]){binary+="0";}else{binary+="1";} Please note 1 is for buy signal and 0 is for sell signal. Binary is the appended binary string which is fed as input to the XCS system. We have deliberately used shorter moving averages (10 and 20) to reduce the lag in the signal and concentrate on the short term trends rather than long term trend. Parabolic SAR: SAR stands for stop and reverse. It was developed by J. Welles Wilder Jr to find trends in market price. It develops dotted line either above or below the security price. The dotted line below the price establish the trailing stop for a long position (generates 19

30 buy signs) and the lines above establish the trailing stop for short position (our System doesn t short and only generates sell sign) Usage in XCS: SAR value greater than current day high of day gives sell sign. if(sar[index]> high){binary+="0";}else{binary+="1";} * Details of SAR Calculation is given in appendix Average Directional Index (ADX): It evaluate strength of current trend, be it up or down. ADX is based on accumulation distribution line. Usage in XCS: Positive and negative direction index (+ DI, -DI ) are used to generate buy and sell sign. if(posdi[index]>negdi[index]){binary+=1;}else {binary+=0;} Commodity Channel Index (CCI): CCI is a typical price based momentum indicator which was developed by Donald Lambert to identify cyclical turns in commodities. Usage in XCS: CCI is band oscillator. Movement above indicates overbought stock and sell signal is given. Similarly movement below -100 gives oversold sign and buy signal is given. Movement between -100 and doesn t give clear sign of buy or sell. In such scenario we have used heuristic that if current CCI is greater than past 5 days moving average of CCI then a buy signal should be given. if(cci[index] >= 100){//over bought binary+=0; }else if (CCI[index] <= -100){//over sold binary +=1; }else if(cci[index] >= CCIMA[index]){// + divergence binary +=1; }else{ binary +=0;} Chaikin Money Flow(CMF): CMF is an oscillator based on accumulation distribution line. Usage in XCS: CMF is bullish when it is positive and bearish when it is negative. if(cmf[index]< 0){binary+="0";}else{binary+="1";} 20

31 MACD: Moving average convergence divergence. It is a centered oscillator that is unique in having both leading and lagging component in it. It is the difference between the 12 day EMA and 26 day EMA of a security. Usage in XCS : A positive macd indicates buy sign and vice versa if(macd[index]> 0){binary+="1";}else{binary+="0";} A nine day Exponential moving average, EMA of MACD acts as trigger line to give buy sells sign if(macd[index] > MACDSignal[index]){binary+="1";}else{binary+="0";} Money Flow Index: MFI is a Momentum indicator similar to RSI. It s a good measure of money flowing into and out of the security. Usage in XCS MFI above 80 indicates overbought stock and gives sell sign and below 20 indicates oversold stock and gives buy sign. In between 20 and 80 there is no clear sign of buy and sold and so we have used positive divergence to create the buy or sell sign. If current MFI is greater than 5 day average of MFI then a buy sign is given. if(mfi[index] >= 80){ binary+=0; }else if (MFI[index] <= 20){ binary +=1; }else if(mfi[index] > MFIMA[index]){ binary+=1; }else{ binary +=0;} On balance Volume (OBV): It s a Volume based oscillator. Usage in XCS A rising bullish OBV line indicates that the smart money is flowing into the stock and shows price uptrend. We have used it as, if the current OBV is greater than past 5 days OBV then a sell sign may be given. 21

32 if (OBV[index]>=OBVMA[index]){binary+=1;}else {binary+=0;} Percentage Price Oscillator (PPO): This oscillator formed by taking difference of longer moving average from shorter moving average of price in percentage form. Usage in XCS: Used in 2 ways-: if(ppo[index]> 0){binary+="1";}else{binary+="0";} if(ppo[index] > PPOSignal[index]){binary+="1";}else{binary+="0";} PPOSignal is found by taking 5 day moving average of PPO. Percentage Volume Oscillator (PVO): Similar to PPO except that instead of price, Volume is used for calculation. Usage in XCS if(pvo[index]> 0){binary+="1";}else{binary+="0";} if(pvo[index] > PVOSignal[index]){binary+="1";}else{binary+="0";} Relative Strength Index (RSI) : RSI is a momentum oscillator which compares the magnitude of a stock s recent gains to the magnitude of its recent losses and turns that information into a number that range from 0 to 100 [18]. Usage in XCS: RSI above 70 and below 30 indicates overbought and oversold condition and gives sell and buy signal respectively. In between 30 and 70 we have used heuristic that if current day RSI is greater than past 5 day average of RSI then a buy sign is given. if(rsi[index] >= 80){ binary+="0"; }else if(rsi[index] <= 20){ binary+="1"; }else if(rsi[index] >= RSIMA[index]){ binary+="1"; }else{ binary+="0"; } 22

33 Stochastic Oscillator: It is a momentum indicator. Usage in XCS Reading below 20 are considered over sold and above 80 are considered over bought. We have used fast percent D in XCS. In between 20 and 80, heuristic similar to RSI is used. if(fastpercentd[index] >= 80){ binary+="0"; }else if(fastpercentd[index] <= 20){ binary+="1"; }else if(fastpercentd[index] >= fastpercentdma[index]){ binary+="1"; }else{ binary+="0"; } Cross over of FastPercentK with respect to fast percent D is also used to generate buy and sell signs. if(fastpercentk[index] > fastpercentd[index]) {binary+="1";}else{binary+="0";} * for details about Stochastic Oscillator calculation, please see the appendix. StochRSI : It is a momentum oscillator wherein Stochastic oscillator is combined with RSI. Usage in XCS: if(stochrsi[index] >= 80){ binary+="0"; }else if(stochrsi[index] <= 20){ binary+="1"; }else if(stochrsi[index] >= stochrsima[index]){ binary+="1"; }else{ binary+="0";} ROC : Rate of change is centered oscillator. It gives percentage price change over the last 20 days. Buy signal generated if ROC is greater than zero. Usage in XCS: if(roc[index]< 0){binary+="0";}else{binary+="1";} 23

34 Williams % R: It s a momentum indicator that works much like the Stochastic Oscillator. Usage in XCS: if(willpercentr[index] >= -20){//overbought binary+="0"; }else{ binary+="1"; } if(( -80 <= willpercentr[index] && willpercentr[index] <= -100)){//oversold binary+="1"; }else{ binary+="0";} Combining different technical indicators and working mechanism of different Agents Few points which we have considered while combining technical indicators are -: 1) Individual indicators in the combination should provide different perspective towards the underlying price or volume movement. Indicators should complement each other instead of moving in unison and generate the same signal [18].For example Chaikin Money Flow (CMF) and Money flow index (MFI) are both price based momentum indicator and provides nearly same information and generates same signal and therefore as such shouldn t be used with each other. 2) It is generally useless to combine more than 5 indicators. Keeping these things in mind and through lot of hit and trial experiments on variety of stocks, we developed 14 set of agents which uses different combination of technical indicators. For example one such combination used for Agent 1 is -: CMF- A non trend following volume indicator to identify buying and selling pressure. RSI - A momentum indicator used to identify potential overbought and oversold levels Moving Average - A trend following indicator to identify the underlying trend in the stock. These indicators have very less in common and complement each other very well. [18] 24

35 Composition of 14 Agents: Name of Agent Composition of Agent Agent 1 Moving Average (10,20), MACD, RSI, CMF Agent 2 Moving Average(10,20), PPO, PVO Agent 3 Moving Average(10,20), Stochastic Oscillator, MACD, CMF Agent 4 SAR, Moving Average(10,20), CMF, Williams %R Agent 5 SAR, ADX, Moving Average, OBV Agent 6 Moving Average (10,20), Williams %R, StochRSI, CMF Agent 7 MACD, ROC, RSI, PVO Agent 8 Moving Average(10,20), CCI, RSI, MACD Agent 9 Moving Average(10,20), CMF, CCI Agent 10 Stochastic Oscillator, ADX, Moving Average(10,20), CMF Agent 11 Moving Average(10,20), PPO, CMF 25

36 Name of Agent Agent 12 Agent 13 Agent 14 Composition of Agent Moving Average(10,20), Stochastic Oscillator, ROC, MFI Stochastic Oscillator, MACD, CMF Stochastic Oscillator, Williams %R, MACD Table1. Composition of different Agents For initial set of experiments only best 7 agents were used. For portfolio management System all 14 sets of Agents were used Advantage and Scope of Improvement of current approach Advantages: 1) All the technical indicators were used in there most basic form. No particular threshold was set for any indicator to generate buy or sell signal. This serves the purpose of minimum priori to XCS system, i.e. providing it with least amount of information so that it can mostly learn from its action and adaptation to the environment. 2) Each individual bit was independent of other bits in the binary string. So no input information was duplicated in any form. 3) Oscillators generally gives sell or buy signal only when they are in over bought or over sold range respectively. By using heuristic we were able to take advantage of upward movement of oscillator and also made sure each oscillator always gives either buy or sell signal. Scope of Improvement in combining technical Indicators 1) There is no single correct or optimum way of combining different technical indicators. A combination which might work for one stock might not work for another one. We have combined technical indicators with our best knowledge. Defining the problem in better way for XCS by combining different technical 26

37 indicators is open ended question and can be explored further. One thing is very clear from our analysis is that the better we combine technical indicators the better we can expect the learning of XCS agents and better could be the returns. 2) Parameter optimization: With individual technical indicators, there are many parameters which can be optimized. In most of the cases we have used widely used parameters. This can be explored further. For example a) For calculating positive and negative Directional index we have used average for 14 days which is widely used. Such short of parameters can be optimized to give best result with maximum number of stocks. b) Moving average and moving volume is taken only for 10 and 20 days. By doing this we have concentrated on short term price or volume movement. By taking longer moving average, longer and more robust trends can be identified. 3.2 Improving the learning of extended Classifier System In the previous version of classifier system, financial forecasting was considered solely as single step problem where each day's Input to the System doesn t have any relation to the next or previous day's inputs. Here successive problem instances were thought of independent of each other and all iterations were treated as independent. This was based on the assumption that Input technical indicator information for one day is completely random and has no relation to any other day. In such scenario the classifier parameters in the current action set [A] are updated only with respect to the immediate reward feedback only. However, a new approach is taken by us where the basic idea is that the state of stock market on any particular day is not completely independent of other days but is affected by how the market has behaved previously in past few days. There is a positive or negative co-relation between the market states of each day. Each classifier represents a particular subset of technical indicator 27

38 information it can act on and the possible action it will take. Keeping this thing in mind, we modeled learning to trade via classifier systems as a Multi Step Reinforcement learning problem where all classifiers in the action set [A] were updated with respect to the immediate reward R plus the estimated discounted future reward (Value function for the next state). Q and Q (λ) are used as multi-step Reinforcement learning algorithm. Each classifier represents a condition it can act upon and the action it will take. So each classifier prediction value can be thought as state action value pair i.e. Q(s,a). Keeping this thing in mind implementing Q learning is trivial in XCS. However one thing has to be kept in mind that due to don t care parameter (#) in the classifiers condition there can be more than one classifier which can match a particular condition and all of them form part of action set. So instead of updating just one classifier in each iteration, we have to update all the classifiers present in action set Classifiers in multi step Reinforcement learning problems A multi step Reinforcement Learning problem poses the additional complication of back propagation of reward in an appropriate manner. Initial complication might arise due to inappropriate reward propagation from inaccurate, young or over generalized classifiers [15]. Q values for a Reinforcement learning problem tell the value of a state action pair. A single classifier in itself is a combination of condition it can act and the action it will take. So the prediction value of a classifier is just like Q values for a particular state in Reinforcement Learning problem. One-step Q-learning, governing equation is This is equivalent to XCS update function for the reward prediction β is the learning rate parameter, γ is the discount rate 28

39 Reward prediction value thus coincide with the Q values Q(s, a). Thus, the prediction array coincides with Q value entries. Without generalization XCS is just a tabular Q learner where each table entry represents a distinct rule. XCS generalizes over states that yield identical Q values with respect to a specified action [15]. A very important point that should be noted for multi step RL as mentioned by Butz [15] is In initial phase of learning the back propagated reward signal is expected to fluctuate significantly. As in RL, XCS is expected to progressively learn starting from those state action combinations that yield actual reinforcement. Once such classifiers are stably represented, back propagation becomes more reliable and the next reward level can be learned accurately and so on. Thus as in Q learning reward will be spread backward starting from the cases that yield the actual reward. Thus XCS learn a generalized representation of the underlying Q function in the problem. [15] Q learning is a temporal difference learning methodology that doesn't require any explicit model of the environment and learns state action value function just by generation of episodes. It is an off policy learning methodology i.e. the policy we follow(behavior policy) is different from the policy we evaluate(or optimize).in our case we follow an ε greedy policy and learn the Q values for optimum policy. This enables early convergence towards optimum Q values. Back up diagram for Q learning is Algorithm for Q learning Figure3: Q-learning: An off-policy TD control algorithm. [14] 29

40 3.2.2 Implementing Q learning in Classifier. The prediction of each classifier in last day action set is updated as P = reward + cons.gamma * maxprediction prediction += cons.beta * (P-prediction) Here maxprediction is the maximum prediction of a classifier in current day action set. Similarly, error property of each classifier in the last action set is updated taking maxprediction into consideration Eligibility trace and Watkins's Q(λ) Q(λ) algorithm is a combination of temporal difference Q learning with eligibility trace to obtain a more general method that may learn more efficiently. In Q leaning, the temporal difference error(td error) is back propagated only to last state visited, whereas in Q(λ) each state is associated with an additional parameter called eligibility trace which indicate degree to which each state is eligible for undergoing learning changes, should a reinforcement event occurs. At each moment we look at current TD error and assign it backward to each prior state according to state eligibility trace at that time. We may think ourselves riding along a stream of states computing TD errors and shouting them back to previously visited states. Figure4:Back ward view of eligibility trace [14] * For details about eligibility trace and Q( λ) algorithm, readers are encouraged to look at chapter 7 of Sutton and Barto book[14]. Watkins's Q(λ) is one form of Q(λ) which can be applied online. It offers the advantage of faster learning. So we applied this to the classifier System to make learning more efficient. 30

41 Watkin s Q(λ) in pseudo code format. Figure5: Tabular version of Watkins's Q (λ) algorithm. [14] For implementing this learning methodology each classifier in the population is associated with an extra parameter named eligibility which shows how much it is eligible for the back propagation of delta error. Watkins Q (λ) is strictly followed as shown in figure 5. Delta error for any day represents error in the prediction.it is calculated as deltaerror = reward + cons.gamma * currpredictionvaluemax - lastpredictionvalue; currpredictionvaluemax is maximum prediction value of a classifier in the current day action set. lastpredictionvalue is the prediction value of the action that was taken yesterday. Eligibility of all the classifier present in the last action set is increased by 1 The delta error is back propagated into the whole classifiers population taking trace decay parameter into consideration. Further, if an exploitation step was taken the eligibility of all the classifier is reduced by a factor of step was taken the eligibility of all the classifier is made zero..however if an exploration 31

42 Chapter 4 4 Experimentation 4.1 FTSE data and Stability of the XCS System FTSE Data We used real data of 90, FTSE100 companies. Company Code Training + Trading Period. Company Name From To ANGLO AMERICAN 5/17/2000 4/29/2008 AAL.L SBRY.L SAINSBURY 5/16/2000 4/29/2008 AV.L AVIVA 5/16/2000 4/29/2008 BT-A.L BT GROUP 11/12/2001 4/29/2008 CBRY.L CADBURY-SCH 5/16/2000 4/29/2008 CCL.L CARNIVAL 4/23/2003 4/29/2008 CAIRN CNE.L ENERGY 2/21/2003 4/29/2008 FGP.L FIRST GROUP 5/16/2000 4/29/2008 FRIENDS PROV 7/10/2001 4/29/2008 FP.L LSE.L LON.STK.EXCH 7/24/2001 4/29/2008 Table2: Details of 10 FTSE 100 Companies * For details of all the 90 FTSE 100 companies used in the experiments please see the appendix. The FTSE 100 companies data was cleaned and preprocessed at Level E Limited and was provided by my supervisor Dr. Sonia Schulenburg Stability of the XCS System. The initial generation of classifier population depends on the random seed and thus Classifiers individual bit can have 0 or 1 or # (don t care symbol) to represent the problem space. Due to this variability is introduced in the system and performance of XCS System varies in different run. A single run of XCS on a particular stock might 32

43 not give clear picture of the learning of the system. To check for the stability of the system we followed following steps-: 1) Run the experiment 100 times to find the mean, max, min and standard deviation of the performance 2) In each run, find the yearly performance of individual agent on the stock. Yearly performance of the XCS agent on a stocks is measured as follows a. After each year, calculate yearly percentage returns as Yearly Percentage = Final Wealth of agents Initial Agents wealth return *100 Initial agent s wealth b. To find the net performance of stock, we can either take arithmetic mean or geometric mean of these yearly percentage returns. We took geometric means of different yearly percentage return as final performance measure for the stock 3) Take the average of yearly performance of all 7 agents to get the performance of an average XCS agent on that particular stock. For our experiment we choose 5 companies which have different volatility characteristics. By volatility, we mean they have different amount of fluctuation in there price pattern. Experiment results Company OM.L LMI.L LLOY.L SVT.L XTA.L Mean Max Min Standard deviation Table3: Experimental results for 100 run on 5 FTSE100 companies. We also ran experiment to find mean for different number of runs like mean for 2, 3,5,7,9,12,16,20,25 runs. 33

44 Mean Performance for different number of runs Performance % Number of runs OML.L LMI.L LLOY.L SVT.L XTA.L Figure6: Mean Performance of 5 companies for different number of runs Observations 1) The system is found to give pretty stable performance as can be seen from low standard deviation for different companies in table 3. 2) Performance is pretty stable and approaches the mean performance after 16 runs. For all our further experiments we ran the system 20 times to see the final performance. All the result henceforth presented are the mean of 20 runs. A very important point to mention here is that XCS is exposed to data only once and learning is completely online. So at any point of time we are not cheating. 4.2 Comparative Study of the 3 different Algorithm The base XCS java code was written by Martin V. Butz and further modified by Matthew Gershoff and Abu ul Hassan during there MSC thesis Setting the parameters for the experiments The first step in comparative study is to find and set the optimum parameters for each algorithm. From performance point of view Classifier Systems are considered very robust in terms of setting of parameters. However we felt it must to at least optimize following 3 parameters-: 1) Initial Exploration rate of all 3 learning methods. 2) Discount rate (gamma) for Q learning. 3) Trace decay parameter for Q(λ) algorithm. 34

45 Setting Initial Exploration Rate To find the optimum parameters, experiments are run on a small subset of stock (5 companies). We experimented with 3 different exploration rates 0.5, 0.2 and For individual stock 7 different XCS agents are trading and there averaged performance is taken for the comparison purpose. For example for company RR.L Only Reward Exploration Exploration Exploration Learning(RR.L) Rate = 0.5 Rate = 0.2 Rate =.02 MetaAgent MetaAgent MetaAgent MetaAgent MetaAgent MetaAgent MetaAgent Agents Average buyandholdagent RadomAgent trendfollowingagent BankAgent Table4: Setting Exploration rate Average shown in Blue is used for comparison purpose. Combined results for different exploration rate Single Step RL Exploration Rate = Exploration Rate = Exploration Rate Companies =.02 RR.L CNE.L BLND.L SABRY AAL.L Average Multi Step RL - Q Learning Exploration Rate = Exploration Rate = Exploration Rate Companies =.02 RR.L CNE.L BLND.L SABRY AAL.L Average

46 Q(λ) Learning Exploration Rate = Exploration Rate = Exploration Rate Companies =.02 RR.L CNE.L BLND.L SABRY AAL.L Average Table5: Combined Results for Different exploration rate For single step RL exploration rate of 0.2 showed better performance than 0.5 and For Q learning also, exploration rate of 0.2 was found best.the performance didn t change much between exploration rate of 0.2 and For Q (λ) exploration rate of 0.2 was found better than 0.5. However, exploration rate of 0.02 also showed nearly same performance. In the end we decided to keep exploration rate to Setting discount rate (gamma) We experimented with 4 different discount rates 0.95, 0.70, 0.50 and 0.2 for different stocks keeping optimize exploration rate of 0.2 and found that keeping high discount rate of 0.95 gave best performance (Which is also default value of gamma in Butz XCS code). Result of one stock is shown below-: RR.L Gamma(0.95) Gamma(0.70) Gamma(0.50) Gamma(0.20) MetaAgent MetaAgent MetaAgent MetaAgent MetaAgent MetaAgent MetaAgent buyandholdagent RadomAgent trendfollowingagent BankAgent Table6: Setting discount rate 36

47 Setting Trace Decay Parameter We experimented with 4 different trace decay parameter 0.1, 0.2, 0.5, 0.8 for Q (λ) algorithm. Combined Result for 9 companies is as shown below Companies Trace Decay(0.1) Trace Decay(0.2) Trace Decay(0.5) Trace Decay(0.8) MKS.L NXT.L OML.L PRU.L PSN.L PSON.L PUB RBL RBS Average Table7: Setting trace decay parameter. Performance of stocks seems pretty robust regarding trace decay parameter. We used trace decay parameter = 0.2 for our further experiments. 4.3 Experimental Results for 3 learning Algorithm We conducted experimented on 86 FTSE companies and compared there average performance against Buy and hold strategy. Company RewardOnly RL Q RL Q(λ) RL B&H RR.L CNE.L PUB.L MKS.L XTA.L LMI.L PSN.L FP.L HBOS.L LSE.L AMEC.L BDEV.L TLW.L CPG.L BB.L

48 Company Reward Only RL Q RL Q(λ) RL B&H TW.L KGF.L CCL.L NXT.L LAND.L STAN.L CPW.L UUL.L SN.L ICI.L RSA.L KEL.L FGP.L ABF.L BA.L AL.L REX.L GSK.L CPI.L IPR.L BP.L REL.L WPP.L SGE.L MRW.L BAY.L PRUL.L RTOL.L BSY.L RTR.L TATE.L LGEN.L SCTN.L DGE.L CBR.L SAB.L RBS.L WOS.L SHP.L BLT.L SSE.L SMIN.L VOD.L TT.L

49 4.3.1 Observations: Company Reward Only RL Q RL Q(λ) RL B&H ANTO.L PSON.L LLOY.L BARC.L IIL.L HSBA.L BLND.L ITV.L CW.L RB.L BATS.L JMAT.L IMT.L LII.L SDRC.L SDR.L DSGI.L AAL.L AZN.L ULVR.L FTSE BTA.L DMGT.L SBRY.L HMSO.L TSCO.L SVT.L Average Table8: Experimental Results for 3 learning methodology 1) Q learning gave 6 % better performance than single step Reinforcement learning. However, Q (λ) performance wasn t better than single step RL. 2) None of the 3 learning strategy was able to beat buy and hold strategy. 3) There were many stocks for which average performance of agents was negative. This indicates that XCS agents for these stocks aren t able to learn the underlying mapping of binary input to the correct action properly. 39

50 4.4 Fault in the previous Reward giving strategy These observations made it clear that the classifiers weren t able to learn properly i.e. there parameters weren t getting updated correctly. We identified that the possible reason for this could be the way reward was judged in the system. The correctness of the classifier s action was decided on the next day price movement of the stock. For example, if today XCS predict that prices are going to go up and it buys the stock, then the reward is given to the classifiers in the action set [A] by checking what happens to the prices the very next day. If the price goes up, which implies it was right to buy the stock, it is given a reward of 1000 else a reward of 0 is given. However, we believe the daily prices of the stock contain a lot of noise, fluctuations and the judgment on the correctness of the action shouldn t be solely done on the next day price movement. For this reason, we changed the reward giving strategy and in the modified system, reward was delayed by 5 days. We took the simple moving average of the next 5 days closing price and on the 6 th day we judged the correctness of the action taken. So if we took a decision of buying the stock on the first day and the average of next 5 days stock price was greater than first day stock price then it means prices were actually in an upward trend and our decision of buying the stock was correct and we further gave a reward of In this way we smoothen the noise in the stock prices and expect it learn properly Experiments with improved delayed reward Strategy We experimented again on all 90 FTSE 100 stocks starting from optimization of the System parameters. 40

51 Setting Initial Exploration Rate We experimented with 3 different exploration rates 0.5, 0.2 and 0.02 Reward Only Learning Exploration Exploration Exploration Companies Rate = 0.5 Rate = 0.2 Rate =.02 RR.L CNE.L BLND.L SABRY AAL.L MKS.L SDR.L Average Q Learning Companies Exploration Rate = 0.5 Exploration Rate = 0.2 Exploration Rate =.02 RR.L CNE.L BLND.L SABRY AAL.L MKS.L SDR.L Average Q(λ) Reinforcement Learning Exploration Exploration Exploration Companies Rate = 0.5 Rate = 0.2 Rate =.02 RR.L CNE.L BLND.L SABRY AAL.L MKS.L SDR.L Average Table9: Setting Exploration Rate For single step RL and Q learning, exploration rate of 0.5 gave the best results. For Q (λ) exploration rate of 0.2 gave the best results. Few things that can be noted from above results are -: 1) Changing exploration rate doesn t affect the performance much. 41

52 2) For Q (λ) small exploration rate gave best performance. This make sense as the eligibility trace is cut off whenever system explores, loosing much of the advantage of Q (λ) algorithm Setting Discount Rate (gamma) We experimented with 3 different discount rates 0.95, 0.50 and 0.25 for different stocks keeping optimize exploration rate of 0.5 and found that keeping low discount rate of 0.25 gives best performance. Company Gamma(0.95) Gamma(0.50) Gamma(0.25) RR.L MKS.L NXT.L OML.L HSBA.L ICI.L MRW.L AVERAGE Table10: Setting discount rate Setting Trace Decay (λ) We experimented with 4 different trace decay parameters and found that keeping high trace decay (0.8) gave best results. Companies Trace Decay(0.1) Trace Decay(0.2) Trace Decay(0.5) Trace Decay(0.8) RR.L MKS.L NXT.L OML.L HSBA.L ICI.L MRW.L Average Table11: Setting trace decay. We kept these optimized parameters for further experimentations. The thing that is very clear from above mentioned results is that Q learning is giving best performance at low discount rate. It means it is tending towards single step reward only RL. Also Q (λ) by setting high trace decay rate is tending towards Q (1) 42

53 Learning. Under such circumstances we don t expect Q (1) or Q (λ) to give better results than single step reward only RL for all companies Results with new delayed reward strategy Results for 90 of FTSE100 companies: We first experimented with single step reward only RL. Agents Average Wealth Old New Reward Reward Strategy Strategy Buy and Hold Standard Deviation (Yearly Performance) Old New Reward Reward Strategy Strategy Company Name AAL.L ABF.L AL.L AMEC.L ANTO.L AV.L AZN.L BA.L BARC.L BATS.L BAY.L BB.L BDEV.L BG.L BLND.L BLT.L BP.L BSY.L BT-A.L CBRY.L CCL.L CNE.L CPG.L CPI.L CPW.L CW.L DGE.L DMGT.L DSGI.L FGP.L FP.L FTSE GSK.L Buy and Hold 43

54 Agents Average Wealth Old New Reward Reward Strategy Strategy Buy and Hold Standard Deviation (Yearly Performance) Old New Reward Reward Strategy Strategy Company Name Buy and Hold HMSO.L HSBA.L ICI.L III.L IMT.L IPR.L ITV.L JMAT.L KEL.L KGF.L LAND.L LGEN.L LII.L LLOY.L LMI.L LSE.L MKS.L MRW.L NXT.L OML.L PRU.L PSN.L PSON.L PUB.L RB.L RBS.L REL.L REX.L RIO.L RR.L RSA.L RTO.L RTR.L SAB.L SBRY.L SCTN.L SDR.L SDRC.L SGE.L SHP.L SMIN.L

55 Agents Average Wealth Old New Reward Reward Strategy Strategy Buy and Hold Standard Deviation (Yearly Performance) Old New Reward Reward Strategy Strategy Company Name Buy and Hold SSE.L STAN.L SVT.L TATE.L TLW.L TSCO.L TT.L TW.L ULVR.L UU.L VOD.L WOS.L WPP.L XTA.L Average Table12: Comparative results for 90 FTSE 100 companies with delayed reward Observations: strategy for single step reward only RL. 1) The new delayed reward strategy gave 263% better performance than old reward strategy. 2) The new delayed reward strategy was also able to beat Buy and hold strategy by 166%. 3) In new reward strategy 88 of 90 companies showed better performance than old reward giving strategy. 4) In new reward strategy 79 of 90 companies (i.e. 88% companies) showed better performance than benchmark buy and hold strategy. 5) The Standard Deviation (risk) for new reward strategy is 23% less than standard deviation for the old reward strategy. 6) Standard Deviation (risk) for new reward strategy is 53% less than standard deviation of buy and hold strategy. 45

56 Let s take few examples to illustrate this Company : DSGI Figure7: DSGI Price Chart The price of DSGI has fluctuated a lot from January 2002 to July With previous approach of immediate reward feedback the very next day, agents gave an average performance of -8.29% annually. Figure8: DSGI, Wealth Chart of agents with old reward strategy Please note few important things from the agent performance with old reward strategy-: 1) The wealth of the agents nearly follows the same pattern as that of stock price. This has been the characteristics of almost all the FTSE 100 Stock. So when the price of stock rises, agent s wealth also rises and when there price falls, agent s wealth also fall. This confirms our initial remark that classifiers are not able to learn the prediction value corresponding to a particular technical indicator binary input properly. 2) None of the agent is able to predict the fall in the price of stock during January 2007 to July

57 Figure9: DSGI, Agent s performance with new delayed reward giving strategy. Points to note for new reward strategy-: 1) With new approach the average performance of Meta Agents rose from -8.29% to 9.85% annually. 2) All the agents were able to predict the fall in stock price from January 2007 to July 2008 and all of them perform better than buy and hold strategy. When Agents predicts fall in the stock price they simply sell the stock and keep the money in bank. Straight lines in the above graph correspond to such behavior. Note similar behavior of another stock LAND.L Figure10: LAND, Price Chart 47

58 Figure11: LAND, Meta Agents wealth chart with old reward strategy. Figure12: LAND, Meta Agents wealth chart with new delayed reward strategy. The average performance of Meta Agents rose from -4.73% to 15.69% annually Experimental Results for all 3 learning algorithm with new delayed reward strategy We also experimented with all 3 learning methodology, single step RL, multi step Q and Q(λ) with the optimum parameters found above. Standard Deviation(Yearly Average Agents Wealth Performance) Only Only Company Name Reward RL Q - RL Q(λ) RL B&H Reward RL Q RL Q(λ) RL B&H AAL.L ABF.L AL.L AMEC.L ANTO.L AV.L AZN.L

59 Average Agents Wealth Standard Deviation Only Only Company Name Reward RL Q - RL Q(λ) RL B&H Reward RL Q RL Q(λ) RL B&H BARC.L BATS.L BAY.L BB.L BDEV.L BG.L BLND.L BLT.L BP.L BSY.L BT-A.L CBRY.L CCL.L CNE.L CPG.L CPI.L CPW.L CW.L DGE.L DMGT.L DSGI.L FGP.L FP.L FTSE GSK.L HBOS.L HMSO.L HSBA.L ICI.L III.L IMT.L IPR.L ITV.L JMAT.L KEL.L KGF.L LAND.L LGEN.L LII.L LLOY.L LMI.L LSE.L

60 Average Agents Wealth Standard Deviation Only Only Company Name Reward RL Q - RL Q(λ) RL B&H Reward RL Q RL Q(λ) RL B&H MRW.L NXT.L OML.L PRU.L PSN.L PSON.L PUB.L RB.L RBS.L REL.L REX.L RIO.L RR.L RSA.L RTO.L RTR.L SAB.L SBRY.L SCTN.L SDR.L SDRC.L SGE.L SHP.L SMIN.L SN.L SSE.L STAN.L SVT.L TATE.L TLW.L TSCO.L TT.L TW.L ULVR.L UU.L VOD.L WOS.L WPP.L XTA.L Average Table13: Results of 90 FTSE100 Companies for different RL with new delayed reward strategy. 50

61 Observations 1) All 3 learning methodology were able to beat buy and hold strategy. Single step RL gave 177% better performance,1step Q learning gave 161% better performance and Q (λ) gave 117% better performance than buy and hold strategy. 2) Both Q learning and Q (λ) performance was less than Single step reward only RL. 3) Standard Deviation (risk) for all three learning algorithm was less than buy and hold strategy. Single step RL showed 52% less standard deviation, 1step Q learning showed 52% less standard deviation and Q (λ) showed 54% less standard deviation Fault in Q (1) learning With our initial assumption that market behavior on different days is corelated, we expected multi step Q learning to perform better than single step reward only RL. However the results were contrary. We found that the reason for this was that currently for any wrong action XCS just gave a reward of zero and didn t penalizes classifier by giving negative reward. Let say for example on a given day XCS can take two actions either sell or buy. It took action sell which turns out to be wrong. We expect the prediction of such classifier with incorrect action, sell to go down. However, in old XCS version for Q (1) RL, with wrong action it will just be given zero reward. Due to this prediction will still go up because of update equation. Here, even if r is zero, Q(s, a) will increase because of γ max Q(St+1,a). We found this approach wrong. To tackle this, either of the below mentioned two things could be done-: 1) If a classifier takes wrong action, give negative reward. The amount of negative reward that should be given is another optimization problem and need careful consideration. OR 51

62 2) After the creation of match set [M] we find which action is winner say 01 (Buy). Make two sets: a) One action set containing all the classifiers who have won (already being done). b) Second a Passive set containing all the classifiers who have lost i.e. there action was not taken. Union of action set and passive set makes match set. In case wrong decision is taken instead of penalizing classifiers in action set (as suggested in point 1), we will give a reward of 1000 to the passive set. This approach makes sense, since if instead of action 01(buy), action 00 (sell) would have been taken then that would have been correct and Passive set classifier would have been rewarded. By working this way we avoid the necessity of giving negative reward Experimentation with passive Set Finding optimum parameters We again started with finding optimum parameters as has been done before. For single step reward only RL and Q learning, exploration rate of 0.5 gave best results on small set of companies and for Q (λ) exploration rate of 0.2 gave best results. Trace decay of 0.2 gave best results for Q (λ). For details of experimental results please see appendix. Discount rate was taken as Comparative Results for FTSE 100 companies using Passive Set. Average Agents Wealth Standard Deviation(Yearly Returns) Only Only Company Name Reward RL Q - RL Q(λ) RL B&H Reward RL Q - RL Q(λ) RL B&H AAL.L ABF.L AL.L AMEC.L ANTO.L AV.L AZN.L BA.L

63 Average Agents Wealth Standard Deviation(Yearly Returns) Only Only Company Name Reward RL Q - RL Q(λ) RL B&H Reward RL Q - RL Q(λ) RL B&H BARC.L BATS.L BAY.L BB.L BDEV.L BG.L BLND.L BP.L BSY.L BT-A.L CBRY.L CCL.L CNE.L CPG.L CPI.L CPW.L CW.L DGE.L DMGT.L DSGI.L FGP.L FP.L FTSE GSK.L HBOS.L HMSO.L HSBA.L ICI.L III.L IMT.L IPR.L ITV.L JMAT.L KEL.L KGF.L LAND.L LGEN.L LII.L LLOY.L LMI.L LSE.L MKS.L

64 Average Agents Wealth Standard Deviation(Yearly Returns) Company Name Only Reward RL Q - RL Q(λ) RL B&H Only Reward RL Q - RL Q(λ) RL B&H MRW.L NXT.L OML.L PRU.L PSN.L PSON.L RB.L RBS.L REL.L REX.L RIO.L RR.L RSA.L RTO.L RTR.L SAB.L SCTN.L SDR.L SDRC.L SGE.L SHP.L SMIN.L SN.L SSE.L STAN.L SVT.L TATE.L TLW.L TSCO.L TT.L TW.L ULVR.L UU.L VOD.L WOS.L WPP.L XTA.L Average Table 14: Comparative Results for FTSE 100 companies using Passive Set. 54

65 Average Agents Wealth Standard Deviation Only Only Learning Type Reward RL Q RL Q(λ) RL B&H Reward RL Q RL Q(λ) RL B&H Active Set Passive Set Optimization Table15: Comparison with Only active set and with additional passive set approach Observations for Passive set. 1) As expected the performance of Q learning increases from 14.93% to 18.70%. However, performance decreases for reward only learning RL and Q (λ) learning. The reason for this is not very clear. 2) Multi step Q learning is better than single step RL for 62 of 90 FTSE 100 companies. 3) Standard Deviation (risk) for Only Reward RL with active set is 28% less the standard deviation for Q, RL. 4) Q learning with passive set approach was found to be best on performance criteria. It gave 18% better than only reward RL. So for further portfolio management only reward RL was used without passive set and Q learning was used with passive set approach. Combined Results single step RL without passive set and multi step Q with passive set. Average Agents Wealth Standard Deviation Company Name Only Reward RL Q RL B&H Only Reward RL Q RL B&H AAL.L ABF.L AL.L AMEC.L ANTO.L AV.L AZN.L BA.L BARC.L

66 Average Agents Wealth Standard Deviation Company Name Only Reward RL Q RL B&H Only Reward RL Q RL B&H BATS.L BAY.L BB.L BDEV.L BG.L BLT.L BP.L BSY.L BT-A.L CBRY.L CCL.L CNE.L CPG.L CPI.L CPW.L CW.L DGE.L DMGT.L DSGI.L FGP.L FP.L FTSE GSK.L HBOS.L HMSO.L HSBA.L ICI.L III.L IMT.L IPR.L ITV.L JMAT.L KEL.L KGF.L LAND.L LGEN.L LII.L LLOY.L LMI.L LSE.L MKS.L

67 Average Agents Wealth Standard Deviation Company Name Only Reward RL Q RL B&H Only Reward RL Q RL B&H MRW.L NXT.L OML.L PRU.L PSN.L PUB.L RB.L RBS.L REL.L REX.L RIO.L RR.L RSA.L RTO.L RTR.L SAB.L SBRY.L SCTN.L SDR.L SDRC.L SGE.L SHP.L SMIN.L SN.L SSE.L STAN.L SVT.L TATE.L TLW.L TSCO.L TT.L TW.L ULVR.L UU.L VOD.L WOS.L WPP.L XTA.L Average Table16: Combined Results single step RL without Passive set and multi step Q Learning with passive set 57

68 Chapter 5 5 Implementation & Experimentation-Portfolio Optimization Most of the discussion till this point, concentrated on the learning of the XCS System and therefore we concentrated on cumulative average performance of all the agents. We haven t looked and used the performance capabilities of single best agents. The agents vary in the type of technical indicators they used to formulate the problem and find the underlying price pattern. Due to this they give good performance for certain stock and bad performance for others. Also, the performance of agents may vary according to whether the stock is in trending period or in trading period. In trending period we expect the combination of lagging indicators like moving average to perform better. In trading periods such indicators will not perform better and combinations of leading technical indicators like oscillators, Stochastic or CMF are expected to perform much better. However, please note while designing the agents we have tried to make them robust so that they give best performance on a variety of stocks. Aim of Portfolio management in XCS is to harness the profit making capabilities of individual agents. We aim to build a subsystem which interact with existing extended Classifier System and do monthly or quarterly portfolio rebalancing and optimization. In this system all the agents will trade. According to some performance measure (either maximum return or maximum sterling ratio for last period) we will see which agent has done best in the last period (month or quarter) and accordingly we will use that agent to trade for the next period (month or quarter). In 58

69 further sections we have explored different strategies like 'taking best 3 or best 5 agents to do the trading','taking top best companies in the portfolio' and 'mean reversal strategy'. 5.1 Implementation For portfolio management we used all 14 set of agents as described in technical analysis section. We needed certain criteria on the basis of which we can judge the performance of the individual agents. The performance measures used were -: 1) Percentage Return from last period- In this we simply find which agents have given us maximum percentage return in the last period (month or quarter) and use that agent to trade for next period. 2) Sterling ratio: It is a risk adjusted return measurement. It is calculated as Sterling ratio = Average Return for the period Maximum draw down for that period Unlike, Sharpe ratio, sterling ratio penalizes volatility only in downward direction. For portfolio performance measurement also, average return from 20 runs, as has been described before is used to gauge the performance of different stocks. Experiments were conducted on 56 FTSE 100 companies (Please note, we wish to do monthly or quarterly rebalancing of portfolio for which it was essential to have same starting date of trading for different stocks. Out of 90 FTSE 100 Companies, only 56 company s data was available from Year 2000 onwards. So, we used only these 56 companies in our portfolio construction and rebalancing). Experiments were conducted on both single step reward only RL and Q learning methodology with the optimum parameters as found before. 59

70 5.2 Portfolio Performance: To measure the overall risk adjusted performance of the portfolio, Sharpe ratio is used. Sharpe ratio was developed by Nobel laureate William F. Sharpe and is calculated by subtracting the risk-free rate such as that of the10-year U.S. Treasury bond (we took standard 5% for this) from the rate of return for a portfolio and dividing the result by the standard deviation of the portfolio returns. Sharpe Ratio = Rp Rf σp Where, Rp = Expected Portfolio return (annual), Rf = Risk Free rate (annual) σp = Portfolio standard deviation A higher Sharpe ratio describes how much extra return we can receive for the extra volatility(risk) endured. While doing monthly portfolio rebalancing modified form of Sharpe ratio formula is used Here, r is monthly portfolio return, f is the monthly risk free rate (taken as 0.42%) and σr is monthly standard deviation in returns. 60

71 5.3 Results Result for one company for which monthly portfolio rebalancing and optimization is done Transaction Date Total Wealth Best Agent Trading 5/2/ Agent1 6/6/ Agent4 7/8/ Agent1 8/7/ Agent1 9/9/ Agent6 10/9/ Agent7 11/8/ Agent12 12/10/ Agent14 1/14/ Agent4 2/13/ Agent4 3/17/ Agent1 4/16/ Agent14 5/21/ Agent2 6/23/ Agent14 Table17: Portfolio Optimization using single best Agent Please note how different agents were used to trade for a particular month depending on there performance for last month. For the first month, May 2002 Agent 1 was chosen as default for trading. For second month June 2002, we compared the performance of all 14 agents for the first month May and found that agent4 gave the best results. We further used this agent to do the trading for second month, June 2002 and so on. 61

72 Portfolio Management Results Figure13: Portfolio Management Results 62