intalus.com/tradesignal-howto Issue 03 STRESS TESTING HOW ROBUST IS YOUR STRATEGY? Proper optimization rather than curve fitting. Powered by A publication of
WATCH VIDEO TO SEE EXAMPLES ON STRESS TESTING HOW ROBUST IS YOUR STRATEGY?. FIND THIS AND MORE TRADESIGNAL HOW TO VIDEOS AT: IN ADDITION CHECK OUR TRADING TIPS VIDEOS AT: www.intalus.com/trading-tips SHARE THIS TRADESIGNAL HOW TO! 2
STRESS TESTING HOW ROBUST IS YOUR STRATEGY? Properly optimization instead of curve fitting. With the right software, testing and optimizing a trading strategy is very easy these days. It is precisely this simplicity, however, that allows users consciously or unconsciously to fall into the curve fitting trap. In this issue we want to show how optimization leads to over-fitting and also how to identify if your strategy is robust by using a stress test. bb Back test and optimization a holy grail? 4 bb Optimization versus curve fitting 5 bb Using the stress test for creating a robust strategy 8 bb The stress test indicator 10 bb Application in practice 13 bb Robustness the key to success 19 3
BACK TEST AND OPTIMIZATION A HOLY GRAIL? Algorithmic trading strategies are based purely on objectives and clear rules: Getting into a position, the stop, exit and the position size are all subject to unique requirements. Besides this, another advantage is that any emotional traps are easily avoidable. The use of quantitative trading strategies also provides a decisive advantage: Any strategy that can be transferred into a source code and as such converted it into a machine language, can be back tested in other words, simulated based on historical data before it is applied to real funds. By using the gained risk and return metrics the characteristics and quality of a strategy can be analyzed in detail and compared with alternatives. Combined with an optimization function, many powerful tools to aid the development of trading strategies are available to the user. SHARE THIS TRADESIGNAL HOW TO! 4
OPTIMIZATION VERSUS CURVE FITTING. Used incorrectly, however, the tools just mentioned can have undesirable consequences, namely curve fitting. The following anecdote, which reiterates a conversation between scientists, Freeman Dyson and Enrico Fermi, describes this phenomenon: It is 1953. Freeman Dyson questions Fermi in Chicago to discuss with him his own results for the meson-proton distribution. Fermi was however visibly unimpressed by the results and asks Dyson, how many selectable parameters he had used for the calculations. Dyson answers that he had used four parameters. Fermi s retort: I remember how my friend John von Neumann used to say, with four parameters I can adjust an elephant, and with five I can make him wiggle his trunk. This quote describes, what is at stake in curve fitting - to over-fitting a trading strategy to the underlying data series. During the optimization process, individual parameters are modified (eg, length of the moving average, stop or target size) in an iterative process until, finally, from a large number of different combinations those are picked, which promise optimal results but only on the basis the specific historical data that was used for the calculation. 5
So before a back test and optimization are performed at all, the trader should have a solid idea for the trading strategy and attempt to define meaningful parameter ranges to be tested. In short, the software should validate a trading idea, not discover it. The more parameters a trading strategy has and the higher the number of combinations tested, the higher the curve fitting effect. The result is that the trading strategy delivers wonderful results in the back test, but fails miserably in real trading. Simulated scientific studies show that the stronger the strategy was adapted to the historical data, the poorer the results in real trading.* * A more detailed study entitled The probability of backtest overfitting can be downloaded for free here: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253 SHARE THIS TRADESIGNAL HOW TO! 6
For this reason, a division of the representative historical market data in at least two sub-periods is necessary. This allows for the development and optimization of a trading strategy based on the period A (In-sample back test) in order to test it on the unknown data-terrain, period B (Out-of-sample back test). The results of the sub-periods should not show any significant deviations. If this were the case, it is highly likely to be evidence of an over-fitted strategy. A special form of this test method in which the optimal parameters of a sub-period for the subsequent out-of-sample period are used,it is called the walk forward test. How you can use it in practice and what features are provided by Tradesignal? We discuss this in greater detail in a future How To issue. 7
USING THE STRESS TEST FOR CREATING A ROBUST STRATEGY. For a trading strategy to work, not only in the back test, but also in real trading, it must be robust. With Tradesignal the back testing and optimization function allows you to check how stable your trading strategy behaves on different markets and in different time frames even using an entire portfolio. The testing of the strategy at different time frames can help identify changes in performance behavior, which would be due to differing market phases. The use of other securities in the back test shows, however, whether the algorithm actually identifies general patterns that can be observed in other underlying assets or if it has been trimmed on specific patterns within the underlying data series. An easy way to test this is to monitor the change in the risk and return characteristics while the parameter values are being modified. The more varied the results, the less robust is the strategy. An extreme negative example is shown in figure 1. The graph shows the profit factor of a trading strategy in response to an entry and exit parameter; red areas represent unprofitable, green for profitable values. As can be seen, of all parameter combinations only one is profitable. If exactly this optimal combination of parameters is the basis for a trading strategy, one thing is inevitable in real trading its failure. SHARE THIS TRADESIGNAL HOW TO! 8
Figure 1: Optimization heatmap Profitable parameter combinations being only available in a small area signal curve fitting in its purest form. 9
THE STRESS TEST INDICATOR. Therefore when performing a back test you always need to make sure, that as many parameter ranges as possible generate satisfactory results around the optimum area. As already mentioned, it should be examined in a stress test; or how the chosen strategy behaves on other markets and different time frames. With the stress test indicator we want to show you a simple and convenient way to perform such a stress test. SHARE THIS TRADESIGNAL HOW TO! 10
Meta: Synopsis("Example indicator that is designed to be the parent of a trading strategy for which the strategy author would like to 'stress test' by varying the indicators input parameters during optimization."), Author("Tradesignal GmbH"); Inputs: Security(SIE, ALVG, DAI, EUR, DAX), // Change the security being tested Period(Minutes1, Minutes5, Minutes60, Daily, Weekly, Monthly) = Daily; // Change the period being tested Instruments: inst( IFF(Security = SIE, "SIEGn.DE", IFF(Security = ALVG, "ALVG.DE", IFF(Security = DAI, "DAIGn.DE", IFF(Security = EUR, "EUR=", IFF(Security = DAX, ".GDAXI", ""))) + " " + IFF(Period = Minutes1, "1m", IFF(Period = Minutes5, "5m", IFF(Period = Minutes60, "60m", IFF(Period = Daily, "daily", IFF(Period = Weekly, "weekly", IFF(Period = Monthly, "monthly", "")))))))) ); DrawBar(Open of inst, High of inst, Low of inst, Close of inst); Figure 2: Stress test indicator Equilla code With the stress test indicator trading strategies can be checked for robustness by defining several securities and time frames as inputs. 11
As you can see, the securities included in this indicator are stored as so-called inline instruments and defined as inputs. Since inputs of indicators can be optimized, this function offers a clever way to test any number of securities against a trading strategy to examine their stability. You can apply the indicator by saving the code above by copying and pasting it into the Equilla editor. In the sample code, the RICs from Thomson Reuters have been used. SHARE THIS TRADESIGNAL HOW TO! 12
APPLICATION IN PRACTICE. An application of the stress test indicator requires the following steps: 01. Copy the Equilla code and paste it in Equilla editor; 02. Modify and expand the securities included in the code if required; 03. Save the indicator on your Tradesginal platform; 04. Insert the indicator into an empty chart; 05. Choose a trading strategy of your choice and add it into the chart (eg Moving Average Crossover Single); 06. Select the toolbar button Optimize a. Select Brute Force b. Parameters to optimize: i. Trading strategy period (5 to 200, step 5); ii. Period of the stress test indicator (eg, monthly, weekly, daily, 60min, 5min); c. Select a data field for the back test 13
Figure 3: Opimization assistant The stress test indicator provides the possibility using different time frames as parameters within an optimization. SHARE THIS TRADESIGNAL HOW TO! 14
After these steps, the user gets an overview that displays the period of the selected parameter of the trading strategy on the x-axis, while the selected time levels are shown on the y-axis. A tabulation (not shown) is also included for detailed analysis. If you want to check the robustness of the strategy for example, measured on profit factor - to analyze the dependence of these two factors, then selecting profit factor is sufficient in the upper input window of the optimizer and one obtains the following chart analysis on next page. 15
Figure 4: Robustness check (parameters versus time frames) The coloured representation of the optimization results shows profitable and unprofitable combinations of two parameters. In this example it is the period of a simple crossover strategy and a total of five different time frames. SHARE THIS TRADESIGNAL HOW TO! 16
Using the color gradation it is possible to detect profitable areas in no time. Green and yellow areas represent profit factor above 1, while orange and red coloured cells represent unprofitable areas. As expected, the exemplarily chosen crossover strategy provides good results on a monthly and weekly time frame (row 1 and 2). On a daily level, positive results can be seen only around the period 200. On 60- and 5-minute-planes (bottom two rows), however, the results are negative across the board. The stress test indicator can be further used to analyze the robustness of a trading strategy as a function of the individual securities and a given period. To do this, not only the period of the SMA, but also the desired securities (parameter selected is Security Strategy (stress test indicator) as the comparison parameter (see step 6b and graph below). A few moments later, the optimization process is complete and the visual presentation just shows all combinations. As you can see, the trading strategy generates positive results basis in all four securities. Particularly high profit factor values are found in the range from 200 to 250 days. 17
Figure 5: Robustness check (parameters versus securities) The colored representation of the optimization results shows profitable and unprofitable combinations of two parameters. In this example, it involves the period value of a simple crossover strategy and four securities. SHARE THIS TRADESIGNAL HOW TO! 18
ROBUSTNESS THE KEY TO SUCCESS. Tradesignal provides all the professional tools, the trader and portfolio manager need for the back test and optimization. The presented stress test indicator provides one of several ways to check the stability and robustness of an algorithmic trading strategy to reduce curve fitting-effects to a minimum. Besides the robustness check, the consideration of realistic costs (commissions, slippage) and the use of a rule-based, dynamic position control size are essential prerequisites for the use of a trading strategy in practice. Finally, we would like to use a famous quote from Albert Einstein that can be transferred very well to Algorithmic Trading. Namely, it provides the easiest and most effective recommendation that needs to be considered when developing quantitative trading strategies: Everything should be made as simple as possible, but no simpler. Albert Einstein Take care, take profit und auf Wiedersehen. 2014 Intalus www.intalus.com Distribution allowed under a Creative Commons Attribution-Noncommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tradesignal is a registered trademark of Tradesignal GmbH. Unauthorized use or misuse is specifically prohibited. All other protected brands and trademarks mentioned in this document conform, without restriction, to the provisions of applicable trademark law and the copyrights of the respective registered owners. 19
Powered by Powered by The ultimate tools for making profitable investment decisions. A publication of intalus.com intalus.com/tradesignal-howto A publication of STRESS TESTING HOW ROBUST IS YOUR STRATEGY? TRADESIGNAL HOW TO. 03 PROFIT FROM THESE TRADESIGNAL HOW TO & VIDEOS TOO. TRADESIGNAL HOW TO. POINT & FIGURE CHARTS FOR BEGINNERS. Issue 01 TRADESIGNAL HOW TO 01 POINT & FIGURE CHARTS FOR BEGINNERS. The ultimate tools for making profitable investment decisions. MORE HOW TO AND VIDEOS HERE WHAT S NEW IN TRADESIGNAL 7.3. New features for more analysis power with ease of use. Issue 02 TRADESIGNAL HOW TO 02 WHAT S NEW IN TRADESIGNAL 7.3. New features for more analysis power with ease of use. SHARE THIS TRADESIGNAL HOW TO! 20
YOU CAN T REACH THE HIGHEST PEAK IN ONE DAY. Get yourself prepared with workshops by Intalus. MORE INFOS AT: www.intalus.com/workshops