Psychological: an excellent predictor of where attention is directed. http://ilab.usc.edu/surprise/
Show this thread
Kullback-Leibler divergence has an enormous number of interpretations and uses: psychological, epistemic, thermodynamic, statistical, computational, geometrical... I am pretty sure I could teach an entire graduate seminar on it.
-
-
-
Epistemic: a normative measure of where you ought to direct your experimental efforts (maximize expected model-breaking) http://www.jstor.org/stable/4623265
Show this thread -
Thermodynamic: a measure of work you can extract from an out-of-equlibrium system as it relaxes to equilibrium.
Show this thread -
Statistical: too many to count, but (e.g.) a measure of the failure of an approximation method.https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained …
Show this thread -
Computational (machine learning): a measure of model inefficiency—the extent to which it retains useless information. https://arxiv.org/abs/1203.3271
Show this thread -
Computational (compression): the extent to which a compression algorithm designed for one system fails when applied to another.
Show this thread -
Geometrical: the (non-metric!) connection when one extends differential geometry to the probability simplex.
Show this thread -
Ooh, biological: the extent to which subsystems co-compute.http://www.mdpi.com/1099-4300/17/4/2432 …
Show this thread -
Another machine learning application: the basic loss function for autoencoders, deep learning, etc. (people call it the "cross-entropy")
Show this thread -
Wait, there's more: algorithmic fairness. How to optimally constrain a prediction algorithm when ensuring compliance with laws on equitable treatment. https://arxiv.org/abs/1412.4643
Show this thread -
Cultural evolution: a metric (we believe) for the study of individual exploration and innovation tasks... https://www.sciencedirect.com/science/article/pii/S0010027716302840 …
Show this thread -
and competitive and collaborative creation and sharing of ideas...http://www.pnas.org/content/115/18/4607.short …
Show this thread -
Someone can explain to me how Kullback-Leibler generalizes to the quantum case—apparently in the case of commuting operators? https://pdfs.semanticscholar.org/30a7/6a44a4f0f882c58bd0b636d6393956258c3f.pdf …
Show this thread -
And if you want to work with generalized entropies and superstatistics (i.e., for coupled systems), it's the special case of the α-Rényi divergence...http://www.pnas.org/content/108/16/6390 …
Show this thread -
For the digital humanists, Kullback-Leibler divergence is related to TFIDF, but with much nicer properties when it comes to coarse-graining. (The most distinctive words have the highest partial-KL when teasing apart documents; stopwords have the lowest)http://www.mdpi.com/1099-4300/15/6/2246 …
Show this thread -
In a very basic sense, Kullback-Leibler divergence is to probability spaces (and thus epistemic states) what the cross-product is to vector spaces.
Show this thread -
Oh, I bet you like mutual information, huh? Well, it's a special case of Kullback-Leibler—the extent to which you're surprised by (arbitrary) correlations between a pair of variables if you believe they're independent.
Show this thread -
It's also (unlike entropy) measure-independent—it behaves well (and in a non-arbitary fashion) for continuous distributions. (A naieve calculation of entropy when you integrate over the uniform distribution is negative!)
Show this thread -
More statistics: it's the underlying justification for the Akiake Information Criterion, used for model selection.
Show this thread -
If people actually want this seminar, we could probably tape it this June through
@ComplexExplorer at@sfiscience. Non-trivial effort but I’d do it sans honorarium if people funded the production costs. Tell them (through the donation button?) http://complexityexplorer.org/about/donateShow this thread -
MORE KL—philosophy of mind. It’s the “free energy” term in the predictive brain account of perception and consciousness. See Andy Clark’s new book orhttps://link.springer.com/article/10.1007%2Fs11229-017-1534-5 …
Show this thread -
Yet more, via
@postquantum, on the quantum case: "if you have some restricted class of operations, then the KL divergence tells you how much of the resource you need (work, entanglement, information) and this measure is unique". https://arxiv.org/abs/quant-ph/0207177 …Show this thread -
Economists have been neglected in this thread, so here's
@itsaguytalking on KL for studying trade under heterogeneous beliefs. http://www.columbia.edu/~ez2197/HowToMeasureDisagreement.pdf …Show this thread -
This is a beautiful paper. A reference to Aumann makes me wonder if KL could be used to measure convergence in "complexity of agreement" problems—https://arxiv.org/abs/cs/0406061
Show this thread -
here, via
@ilmarb, is a talk by@johncarlosbaez on KL in biology—https://www.youtube.com/watch?v=IKetDJof8pk …Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.