Hey everyone. Been lurking for a while, but this is my first time posting something technical. I figured this place is one of the few where folks might actually care about a falsification report instead of just the final results.
So here’s the deal.
In the past six months, I’ve put five different families of trading strategies through their paces using the same protocol: a walk-forward approach combined with a held-out validation test. The held-out test checks if a strategy like "choosing the parameter combination that performed best in the oldest fold" actually outperforms random choices in later folds. A lot of backtests don’t hold up here since the supposed advantage disappears once you factor in the hidden costs of selecting parameters based on prior data.
Four out of the five strategies flopped. Only one managed to hold its ground. I’m sharing this successful strategy and the complete report so that people can critique the methodology.
The four that didn’t make it:
1. Signal-based strategies on USDC majors (using MACD/RSI/ADX with six different setups). Random pair selection outperformed the strategy in 64.5% of tests.
2. The same approach on 4-hour and daily charts. There was a 73-131% decay out-of-sample at the 4-hour mark. The daily had only 4 trades over 4 years, so it’s too sparse to draw any solid conclusions.
3. A CNN walk-forward strategy with nine different architectures. All nine failed outside of the training-adjacent fold. The best one lost $423 out of a $2k notional amount on the second fold.
4. Stat-arbitrage using cointegration on nine majors and 36 pairs, rolling Engle-Granger with a year-long window, z-score reversion, and 60 bps fees. The findings were:
Rule "fold 0 positive" (8 pairs): -5.86% on out-of-sample folds.
Random 1000 trials average: -4.03%.
Oracle (impossible to achieve retrospectively): +1.07%.
Random choices beat the rule in 93.8% of tests. The rule was even negatively predictive.
And the one that actually worked is MVRV.
MVRV Analysis on BTC: 8-Year Walk-Forward Test with Validation
0 replies 130 views
?Reply
Sign in to reply to this topic