@senanurcetin3
Senanur ÇetinDeveloper▲6
Data Scientist & AI Engineer, Nexus
58m ago · AI Engineering · Istanbul

Market forecasting on 804M rows: my hold-out was a lucky stretch

For the MSCapital Kaggle competition I built a short-horizon return forecaster on market microstructure data: 804.5M raw rows of orders, trades and quotes, reduced in BigQuery to 1.26M samples so the model could train on a 16 GB laptop.

Validation used five chronological periods with a one-month embargo, LightGBM and XGBoost base models, and a blended ensemble.

The numbers (cosine score):

  • Walk-forward CV mean: 0.141
  • Untouched hold-out, months 65 to 70: 0.152
  • Leaderboard: 0.128 for the first submission, 0.129 for the shipped ensemble

The gap taught me more than the scores. Scoring a fixed model on every period showed difficulty swinging from 0.117 to 0.148, and my hold-out sat at the 83rd percentile. It was a lucky stretch, not proof that the model was better. The CV mean is optimistic too, because the blend weights were fitted on the same folds.

I tested six hypotheses for the leaderboard gap: one confirmed, one real but smaller than expected, four falsified. It's research only, with no trading claims.

How do you check whether your hold-out period was simply an easy one?

Code: github.com/senanurcetin/ms-capital-market-forecasting

ForecastingEvalsMachine Learning

Comments

No comments yet
Start the conversation.
Log in to join the thread.