Why I don't use OOS when testing trading systems (Out Of Sample)? Because there is absolutely no need for it. But I'll start with when OOS is needed.
1) when searching for a system by automatic selection of indicators and parameters.
There are various programs, who themselves select the system for any instrument by enumerating indicators, mathematical ways, etc.. For example, Adaptrade Builder, ForexStrategyBuilder и др. It is clear that in a certain section of the chart, you can pick up something, in order to build more or less smooth upward equity. If you still apply a bunch of filters and optimize the parameters, then equity on some part of the chart will be smooth to the point of impossibility. But immediately after IS (in Sample) completely different patterns will begin to operate and, with great probability, the system will not pass the OOS test, what is a positive thing for a trader. And it would be nice that ALL variants of systems would not pass OOS, but there is such a system, which will completely accidentally pass the test and the trader will have false hope and he will put such a system into real trading. What is it fraught with, all current traders probably know. That's why, inference, what use OOS in this case, what do not apply, the result will be the same. And all because in this way you cannot build a trading system. Well, perhaps only for one specific phase of the market. And the phases change without warning.
2) when searching for a system by neural network programs.
These programs usually yield very smooth upward equity., well, just grail. And to debunk them, just need OOS. But again, we fall into the trap, так как есть вероятность что OOS совершенно случайно продолжит восходящую эквити и trader поверит в найденный грааль. Better not believe, because OOS will hurt him a lot.
3) when checking trading ideas from the bulldozer, Sort of, I'll check it out, I'll trade by stochastic.
Most likely, this approach will give random equity and the same OOS, but there is a possibility that the inspector will fall entirely into one phase of the market (for example a violent trend) and the tested IS and OOS will both be located in this phase. The result will be, as if, Confirmed, but in real life, when the trend changes to flat, the system will disappoint the trader. I.e, and in this case OOS gives false hope.
So how to be?
It's that simple. As everyone knows, initially there should not be an enumeration of indicators, and a trading idea, that is, the observed pattern of the market, or as many call it market inefficiency. Noticed her, coded and tested in a tester over a long period of time, to include all phases of the market — trend, flat, high volatility, Low, wide range, narrow range, etc.. In this case, the trading idea in the head is IS, and checking it in the backtester will be considered OOS.
If very smooth equity appears as a result of the backtest, no drawdowns at all and, at least, unprofitable months, then you need to throw out such a system. Or rather not to throw it away, and check the code for a peek into the future and transactions for the realism of their commission. There will definitely be something wrong. There are no such inefficiencies in the markets. Smooth equity without drawdowns under 45 degrees maybe, for example, in the following cases:
1) Testing over a huge period of time, for example with 1950 year on 2016. Then the drawdown will simply not be visible visually on such a compressed chart.. They are not visible, but they are. Test years, at least for 5 and drawdowns are immediately visualized.
2) With a very small average income per trade, comparable to the spread. When to consider commissions and slippage, equity will immediately turn into a losing one.
That's why, the equity of the real earning system must be realistic — with drawdowns, sometimes prolonged, even with unprofitable years. And now, if we try to find a realistic, having an advantage over the market, trading system to apply OOS, then we can simply fall into a new trap. For the OOS period, the planned, natural drawdown….. and we will reject our real grail. This will be, really, catastrophe, after all, grails in modern markets are one or two and missed….. :)
That's why, in this case, the use of OOS carries a high risk. In my opinion, it is enough that the trading idea (IS), that appeared in my head was confirmed in the backtest (EAST). And then, already received over a long period of time, Equity, investigate for drawdowns, annual income, time to update new equity peaks, etc.. If the idea of the system was originally, then there should be an understanding why, for example, there was a drawdown at a certain phase of the market, why there was a sharp rise in equity in another phase of the market, etc.. — in general, all this needs to be investigated in detail on the resulting equity. Get an understanding of the change in equity at any time, and then OOS will simply become unnecessary.
This is why I am not using OOS :)
