Mechanism
Different price feeds have different highs/lows. Different sessions have different bars. Backtests can silently benefit from look-ahead bias, survivorship, and curve-fitting.
If tiny parameter changes break performance, you likely optimized noise.
Real data is messy. Your system must survive mess.
Model risk: small parameter changes that flip your results are a warning, not a feature. Robustness is a survival requirement.
- Look for wide plateaus, not sharp peaks.
- Measure drawdown shape, not just final equity.
- Assume the future will be different in the exact way that hurts.
How it kills accounts
Backtest on ideal data → go live → signals misfire → confusion → tweaks → overfit → abandonment.
How it kills accounts:
- Model works on clean history.
- Regime changes and execution friction increases.
- Performance decays slowly, so you rationalize.
- You optimize parameters instead of reducing risk.
- Drawdown becomes the teacher.
Rule that survives
Document data assumptions (feed, session, roll rules).
Test robustness across feeds and parameter ranges.
If it’s fragile, it’s not an edge.
Rule that survives:
- Stress test tails and execution, not averages.
- Prefer robust plateaus over optimized peaks.
- When performance decays, reduce risk before “fixing” the model.
Example archetype
Your breakout strategy works perfectly on one broker feed. On another feed it triggers differently and loses. The ‘edge’ was partly the feed.
Tell: if small parameter tweaks flip your results, your model is fragile.
Deep dive
Robustness beats brilliance
Finance rewards robustness. Fragile elegance dies when reality arrives.
Glossary: overfitting, look-ahead bias.
Field checklist
- Stress-test the tails. The worst days define survival.
- Use variable spreads and slippage in testing.
- Prefer robust plateaus over optimized peaks.
- Look at drawdown shape, not only profit.
- If a tiny parameter change breaks the model, the model is fragile.