given a table of transition probabilities we can estimate P(sequence) by product of P(each); but how best to normalise for length? #obvious?
-
-
@avibryant not sure of the rationale; I'll reply with a gist when kids are asleep :) -
@avibryant https://gist.github.com/matpalm/729c179b203c39a8c646 … as promised :) though maybe just some kind of average will be enough. -
@mat_kelcey How does normalization by sentence length break down here? Normalizing your log likelihood by len(sentence) should work right? -
@cdubhland yeah, works well, not sure why I hadn't just tried it :) ended up using mean(trans_prob^1.5) , the ^ give a bump to high P ones
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.