macrocephalopod

This is a really interesting question, and of massive practical importance if you are managing systematic strategies. To generalize a bit -- someone tells you they have a strategy with a Sharpe of S. What questions should you be asking about the strategy to verify that it's real?https://twitter.com/o_wutang/status/1359686542466445312 …

First let's talk about overfitting. A simple model of the research process is to imagine quants doing a big grid search over hundreds of independent parameters and choosing the best result to present (this sounds dumb but is ... distressingly close to the truth)

If you run N independent strategies on T years of data, the Sharpe of the best strategy will be about S_max = sqrt(2 x log(N) / sqrt(T)) -- inverting that shows that you need to test N = exp(0.5 x T x S^2) parameters to find a strategy with this Sharpe in backtest.

For example, testing ~25 strategies over a one year period will find one with a Sharpe of 2+, and testing ~500 strategies will find one with a Sharpe of 3+

So it's clear that the length of the backtest, and the size of the search space, are absolutely critical things to know to assess the backtest. With a short time period and a large search space you can find a backtest which is arbitrarily good