macrocephalopod

This is a really interesting question, and of massive practical importance if you are managing systematic strategies. To generalize a bit -- someone tells you they have a strategy with a Sharpe of S. What questions should you be asking about the strategy to verify that it's real?https://twitter.com/o_wutang/status/1359686542466445312 …

First let's talk about overfitting. A simple model of the research process is to imagine quants doing a big grid search over hundreds of independent parameters and choosing the best result to present (this sounds dumb but is ... distressingly close to the truth)

If you run N independent strategies on T years of data, the Sharpe of the best strategy will be about S_max = sqrt(2 x log(N) / sqrt(T)) -- inverting that shows that you need to test N = exp(0.5 x T x S^2) parameters to find a strategy with this Sharpe in backtest.

For example, testing ~25 strategies over a one year period will find one with a Sharpe of 2+, and testing ~500 strategies will find one with a Sharpe of 3+

So it's clear that the length of the backtest, and the size of the search space, are absolutely critical things to know to assess the backtest. With a short time period and a large search space you can find a backtest which is arbitrarily good

You also want to know about the assumptions being made, particularly about when the data is available to trade on, how quickly you can execute trades, and what slippage you expect to experience (in terms of fill rate and implementation shortfall)

The simplest explanation for a Sharpe 5 backtest is look-ahead. Many variants of this but essentially it means the backtest assumes some data is available before it would be in reality, i.e. the strategy is allowed to see the future.

Another mistake is to assume that you can observe a signal and instantly trade on it at the observed market price. In reality there is a lag between observing a signal and trading on it, and the market moves in that time. If the signal is good, the move is normally against you.