Market forecasting on 804M rows: my hold-out was a lucky stretch
For the MSCapital Kaggle competition I built a short-horizon return forecaster on market microstructure data: 804.5M raw rows of orders, trades and quotes, reduced in BigQuery to 1.26M samples so the model could train on a 16 GB laptop.
Validation used five chronological periods with a one-month embargo, LightGBM and XGBoost base models, and a blended ensemble.
The numbers (cosine score):
- Walk-forward CV mean: 0.141
- Untouched hold-out, months 65 to 70: 0.152
- Leaderboard: 0.128 for the first submission, 0.129 for the shipped ensemble
The gap taught me more than the scores. Scoring a fixed model on every period showed difficulty swinging from 0.117 to 0.148, and my hold-out sat at the 83rd percentile. It was a lucky stretch, not proof that the model was better. The CV mean is optimistic too, because the blend weights were fitted on the same folds.
I tested six hypotheses for the leaderboard gap: one confirmed, one real but smaller than expected, four falsified. It's research only, with no trading claims.
How do you check whether your hold-out period was simply an easy one?






Comments
0 comments