Sorry one last question before I go do work. Is there a computationally efficient approach to doing pairwise correlation of a large amount of time series at discrete intervals (so given a time range T which can be broken into {t, t+e} intervals)?
Even with overlapping intervals it is still embarrassingly parallel. If you are finding it very slow then honestly just get more cpus working on it. Algorithmic improvements will only give a small advantage. 100 cpus will get you a 100x speed up.