Generating sequences from a language model using Ancestral, Top-K, or Nucleus Sampling? Consider using Tail Free Sampling instead! https://trentbrick.github.io/Tail-Free-Sampling/ …
Thread
-
Prikaži ovu nit
-
Tail Free Sampling tries to ensure you sample diverse and high quality sequences by finding where the probability distribution for the next token to be generated plateaus. Here is an example with different hyperparameters: 0.9 (green) and 0.95 (blue) tend to work wellpic.twitter.com/6NslQvQqlg
1 reply 0 proslijeđenih tweetova 2 korisnika označavaju da im se sviđaPrikaži ovu nit -
I argue this approach explicitly finds the set of “replaceable” tokens for a particular context and that languages (including that of biology) have this replaceability property. If you’re interested please reach out and/or give me feedback.
1 reply 0 proslijeđenih tweetova 2 korisnika označavaju da im se sviđaPrikaži ovu nit -
This work is currently a blog post rather than a paper because I have been unsuccessful in empirically validating Tail Free Sampling against Top-K and Nucleus Sampling.
0 proslijeđenih tweetova 2 korisnika označavaju da im se sviđaPrikaži ovu nit
(neither Top-K or Nucleus Sampling have done empirical validation before, probably for the very reasons why I am finding it difficult!) More details on what I have tried and why this validation is hard are in the blog post :) https://trentbrick.github.io/Tail-Free-Sampling/ …
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.