let’s try it! @openAI want to test your full model on those?
@JeffDean want to try? @etzioni @YejinChoinka @ylecun ?
would love to see any general-purpose model (not directly tailored to these examples) that can approach grade-school performance.
open challenge to all.https://twitter.com/sterlingcrispin/status/1188286346487455744 …
-
Any multiple choice dataset can be evaluated generatively, like we did for Cosmos QA (https://arxiv.org/abs/1909.00277 ), but the reason why generative evaluation is not the mainstream yet is because the field does not yet know how to evaluate the system output automatically.
true. but this is where the benchmarks kowtow to the needs of the field, when in fact the field should move to the benchmarks that more deeply probe comprehension, even if they don’t. fit some preconceived testing paradigm.
