Prof. Anima Anandkumar

Sure. But for practical purposes (e.g. trading), it’s ok to assume coin flips are I.i.d. For practical purposes we also want to model rare, one-off events; but how should we go about that? Using the same tools we have to model coins seems inappropriate here

You can think of this in the language of scoring rules or decision theory where the probability is a probabilistic forecast. Essentially, the [0, 1) interval is your forecaster’s action space.

Then, you can ask the same forecaster to give you probabilistic forecasts of multiple events, and then, say, measure it’s track record over multiple forecasts using a scoring rule S(q_i, x_i) where q_i is the forecast (probability) and x_i is the observed outcome

The scoring rule S tells you: how much penalty do I get if I published a forecast 0.3 and I then actually observe a positive outcome. The log loss is a common choice.

In the i.i.d. Case, this gives you your empirical risk, or likelihood. But you can use this to compare the track record of two forecasters over multiple forecasted events.

Giving a probabilistic forecast makes sense even without assuming that the world is indeed random. You can say I’m giving you a 0.3 probability of rain today rather than tell you it won’t be raining just in case it eventually is raining, and I don’t want to appear very stupid.

So I’d suggest reading up on scoring rules (I know) only because it gives you a language to talk/think about this. Also, you can always invoke @maosbot to give you the full Bayesian dogmatic answer.

Thanks @fhuszar ! I think your comments assume that the forecasters can be “calibrated” a la scoring rule by making predictions on multiple events. That assumes that there is some mutual information to be had between likelihoods of past and future events?