if too many are chosen, it leads to a kind of "overfitting" that is too laser focused e.g. there are billions of stars but if you just randomly point a laser into the sky it's far more likely you'll hit darkness than a starhttps://twitter.com/a_yawning_cat/status/1390158307273306113?s=20 …
just saw github copilot described as fancy markov chain autocomplete which works for general usage as well
-
-
thinking about this more there may be a weakness in this model b/c language isn’t a static state independent of the reader it makes technical sense as an internal model just like u can consider the input/output a long alphanumeric string
Show this thread -
but the actually useful understanding is a much smaller space. blank spaces demarcate words so a string of words is better. words express ideas so input/output as an “idea” is even better
Show this thread -
gpt-3 after all is trained on “ideas of the internet” but this makes gpt-3 feel too much like a search engine. you put in an input to specify some kind of unique state, and then it gives you adjacent states like a librarian fetching a book
Show this thread -
in practice, what is expressed is not a specific idea with an adjacency of other ideas but a STRUCTURE or a function that can take in new parameters and pump out entirely new results
Show this thread -
the similar structure in regular language is the analogy or metaphor. once you hear a good analogy, it's like you have a whole new tool to think with just like having a whole new function in codehttps://twitter.com/a_yawning_cat/status/1390158309202677763?s=20 …
Show this thread -
that is... once you create a "woo portal" gpt-3 can act as a stabilizer that keeps the portal open that would allow my army of demo... er... angels of lights and goodness to step throughhttps://twitter.com/a_yawning_cat/status/1341959274964996096?s=20 …
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.