#033: 5 PIECES OF ADVICE ON HOW TO REDUCE OVERFITTING
Over-optimization (overfitting) is the biggest threat of AOS strategies and also a topic of never ending discussions that don’t have any simple, general, solution. In today’s article, I would like to share with you 5 pieces of advice, how I myself try to minimalize the danger of overfitting.
1. Build your systems using 40-50% of all historical data you have at disposal. Leave the rest for robustness tests
When I create my own strategies, I follow the rule to save as much data as possible. I try not to waste any more data than necessary to create a system, but leave some of them for robustness testing.Because of this reason, for building a system I use just part of all historical data that are available. Usually, I have 10-12 years of historical data and for the building I use only 3-4 years. This period is different every time I build a new strategy (sometimes I use 2-5 years, sometimes 7-10). The reason is simple - the less data I use for creating the strategy itself, the lower is the risk of overfitting.The rest of the data I leave for my robustness tests. If the strategy looks viable on the small, limited sample of data, I start with the robustness tests using all data. Using this method I can be sure that the strategy is tested on the data that were not used for the building of this strategy - this makes the testing more objective and I reduce the risk of overfitting.Of course, this approach gives me a lot of “garbage”, i.e. lot of potentially interesting strategies that fail robustness tests. But this is part of developing high-quality ATS - just by being patient and disciplined, you are getting an edge over other traders, because the majority of them doesn’t have these qualities and tends to look for shortcuts that lead to finding a strategy faster, however, the quality is questionable.Finding a high-quality and robust strategy needs time and patience. And the less data we use for the strategy building, the lower is the risk of overfitting.
2. A good strategy should work in other markets in the same category
The ideal strategy is the one that works also in other, different, incoherent markets. Finding such a “gem” is really extremely demanding and I am not far from the truth when I say that even if you search really intensively, it is a success to find 1 strategy per year that meets this criterion.Since we need to diversify our ATS portfolio, for practical reasons we cannot move forward at such a slow pace. And it is sometimes better to “loose the rules” a little bit and accept strategies that pass all robustness tests in other markets in the same category. Finding these strategies is much more realistic, and the frequency of finding a strategy like this one gets to the acceptable pace.For example, I try a strategy created for TF market to work also in ES, YM, EMD and preferably also in NQ, i.e. to work roughly in all markets from an index category. Just as a strategy created for TY market, I want it to perform well in US and FV markets - i.e. to perform well in all markets in the bonds category.Before we move to the next point, let me make one important note - sometimes it happens that during the strategy verification in other markets, you will encounter a situation when the strategy works even better in another market than the one it was created for. In that case, you will be tempted to trade the strategy in this market. Don’t do it. It is just another form of optimization and these small optimizations sum up and lead to one big overfitting. Simply trade the strategy in the market it was created for, or trade it in all markets in the category. Learn to minimize the attempts to pick the best from the range of acceptable results as this already leads to overfitting.
3. Never amend a strategy after you have finished robustness tests and chosen the parameters for live trading
When the strategy passes all robustness tests and you choose the right parameters for live trading, don’t amend the strategy any more!In the past, I witnessed a behaviour when some traders tried to further “improve” the strategy or slightly tune up the parameters. This is a high grade of overfitting after which, the strategy will almost certainly fail. Yes, you will have nice backtesting equity, but the results from live trading will be one big disappointment.You need to approach the whole optimization process sensitively and at every moment make as little changes as possible. Changing repeatedly the parameters in order to find the best ones leads to significant overfitting. Accept the fact that once you choose the parameters for your live trading, they are the final ones and you should not change them.If you have more than one well-performing set of parameters, you should prefer the average one to the best one. The best results tend to be overfitted. I would either start live trading all sets of available parameters or I would choose the average one.
4. Don't skip the paper trading period
One of the most simple and the most effective ways how to eliminate potentially over-optimized strategy is the paper trading period. When you complete the strategy, run it for 3 months on the simulated account and watch, if the strategy follows your expectations (some of the traders I know go even for 6 months). You will not get any advantage by skipping this period or by trying to find shortcuts. The only really unseen data (the real out of sample data) are the ones that don’t exist yet. And that is exactly the reason why no final verification test can compete with the paper trading period when we test the strategy on live data. Being impatient doesn’t lead anywhere. The best way to move forward is the patience and discipline.
5. Don't use too low time frames (to reduce the “distortion”)
According to my experience, I can say that the lower time frame you use to create a strategy, the more distortions are contained in the charts. How does it relate to overfitting? Quite simply. A nice equity on low time frame is rather the result of overfitting than of an edge. Higher time frames contain less distortion, therefore, there is a better assumption that there is a real edge behind the equity that we see.Personally, I use several time frames, but never lower than a 10-minute. For day trading strategies I generally use 15, 30, 45, but even a 60-minute time frames. For my swing strategies, I go even higher - to 80, 120, 140, 180, 200 or even up to 240-minute. I have several strategies across several time frames - which I consider another way of diversification.One final note to time frames - verify your final strategy on several time frames. You might find a time frame which gives you better results than the original one, don’t trade it live - it is another way of over-optimization. Use the time frame you have created the strategy for and in the worst case, choose the one with average results.Everything I have described above seems to be small things, but when you put them together, it is a big step to overfitting reduction.
Happy Trading!
Tomas
